Assistant with gpt-4o and gpt-4o-mini may call unsupported tool 'browser' and throw exception

[kunerzzz]

Confirm this is an issue with the Python library and not an underlying OpenAI API

• This is an issue with the Python library

Describe the bug

When calling the Assistant API and selecting the gpt-4o or gpt-4o-mini model, the model may attempt to call an unsupported browser tool. The Python SDK does not define this type of ToolCall , resulting in an exception being thrown.

File ""assistant.py"", line 62, in <module>
    stream.until_done()
  File ""/usr/local/lib/python3.7/site-packages/openai/lib/streaming/_assistants.py"", line 102, in until_done
    consume_sync_iterator(self)
  File ""/usr/local/lib/python3.7/site-packages/openai/_utils/_streams.py"", line 6, in consume_sync_iterator
    for _ in iterator:
  File ""/usr/local/lib/python3.7/site-packages/openai/lib/streaming/_assistants.py"", line 69, in __iter__
    for item in self._iterator:
  File ""/usr/local/lib/python3.7/site-packages/openai/lib/streaming/_assistants.py"", line 406, in __stream__
    self._emit_sse_event(event)
  File ""/usr/local/lib/python3.7/site-packages/openai/lib/streaming/_assistants.py"", line 267, in _emit_sse_event
    run_step_snapshots=self.__run_step_snapshots,
  File ""/usr/local/lib/python3.7/site-packages/openai/lib/streaming/_assistants.py"", line 913, in accumulate_run_step
    data.delta.model_dump(exclude_unset=True),
AttributeError: 'dict' object has no attribute 'model_dump'

The root cause of this issue is that the API returns unexpected content. However, the SDK should be able to handle this scenario gracefully.

To Reproduce

1. Ask assistant to use browser tool but only give it code interpreter.

with openai.beta.threads.create_and_run_stream(
    assistant_id='an assistant only supports Code Interpreter',
    model='gpt-4o-mini',
    instructions="Use browser tool first to answer the user's question",
    thread= {
        'messages': [{'role': 'user', 'content': 'Who is Tom'}]
    },
    tool_choice='required',
) as stream:
    stream.until_done()

This issue can occur under normal usage scenarios, but it is consistently reproducible with the specified configuration.

Code snippets

No response

OS

Linux

Python version

Python 3.7.4

Library version

openai v1.36.0

[RobertCraigie]

Hi @kunerzzz I can't reproduce this, could you share a full example script?

from openai import OpenAI

client = OpenAI()

assistant = client.beta.assistants.create(
    name="Math Tutor",
    instructions="You are a personal math tutor. Write and run code to answer math questions.",
    tools=[{"type": "code_interpreter"}],
    model="gpt-4o-mini",
)

with client.beta.threads.create_and_run_stream(
    assistant_id=assistant.id,
    model="gpt-4o-mini",
    instructions="Use browser tool first to answer the user's question",
    thread={"messages": [{"role": "user", "content": "Who is Tom"}]},
    tool_choice="required",
) as stream:
    stream.until_done()

this passes with the only output being a pydantic warning

/openai-python/.venv/lib/python3.9/site-packages/pydantic/main.py:347: UserWarning: Pydantic serializer warnings:
  Expected `Union[RunStepDeltaMessageDelta, ToolCallDeltaObject]` but got `ToolCallDeltaObject` - serialized value may not be as expected
  Expected `Union[CodeInterpreterToolCallDelta, FileSearchToolCallDelta, FunctionToolCallDelta]` but got `CodeInterpreterToolCallDelta` - serialized value may not be as expected
  return self.__pydantic_serializer__.to_python(

[kunerzzz]

Hi @RobertCraigie,

I just checked the version of pydantic and found that in the problematic environment, version 2.0.0 is installed.
In this version, the browser ToolCall's RunStepDelta is deserialized as a dict, whereas in the newer version of pydantic, it is deserialized as <class 'openai.types.beta.threads.runs.run_step_delta.RunStepDelta'> and triggers a warning.

Updating pydantic seems to solve my problem.

[kunerzzz]

@RobertCraigie Sorry, there are still some issue

client = OpenAI()

class EventHandler(AssistantEventHandler):
    def __init__(self) -> None:
        super().__init__()

    @override
    def on_event(self, event: AssistantStreamEvent) -> None:
        if isinstance(event, ThreadRunStepCompleted):
            if event.data.type == 'tool_calls':
                for tool_call in event.data.step_details.tool_calls:
                    # type(tool_call) is <class 'dict'>
                    if tool_call.type == 'code_interpreter':
                        pass
                    elif tool_call.type == 'file_search':
                        pass

    @override
    def _emit_sse_event(self, event: AssistantStreamEvent) -> None:
        if event.event == "thread.run.step.delta":
            print(f'{type(event)} {type(event.data)} {type(event.data.delta)} {str(event.data.delta)}')
        super()._emit_sse_event(event)


client = OpenAI()

assistant = client.beta.assistants.create(
    name="Math Tutor",
    instructions="You are a personal math tutor. Write and run code to answer math questions.",
    tools=[{"type": "code_interpreter"}],
    model="gpt-4o-mini",
)

with client.beta.threads.create_and_run_stream(
    assistant_id=assistant.id,
    model="gpt-4o-mini",
    instructions="Use browser tool first to answer the user's question",
    thread={"messages": [{"role": "user", "content": "Who is Tom"}]},
    tool_choice="required",
    event_handler=EventHandler()
) as stream:
    stream.until_done()

Output:

<class 'openai.types.beta.assistant_stream_event.ThreadRunStepDelta'> <class 'openai.types.beta.threads.runs.run_step_delta_event.RunStepDeltaEvent'> <class 'openai.types.beta.threads.runs.run_step_delta.RunStepDelta'> 
RunStepDelta(step_details=ToolCallDeltaObject(type='tool_calls', tool_calls=[CodeInterpreterToolCallDelta(index=0, type='browser', id='call_6GeU3til5JdXMxUCRmjLZK0L', code_interpreter=None, browser={})]))
/lib/python3.12/site-packages/pydantic/main.py:308: UserWarning: Pydantic serializer warnings:
  Expected `Union[RunStepDeltaMessageDelta, ToolCallDeltaObject]` but got `ToolCallDeltaObject` - serialized value may not be as expected
  Expected `Union[CodeInterpreterToolCallDelta, FileSearchToolCallDelta, FunctionToolCallDelta]` but got `CodeInterpreterToolCallDelta` - serialized value may not be as expected
  return self.__pydantic_serializer__.to_python(
Traceback (most recent call last):
  File "demo.py", line 51, in <module>
    stream.until_done()
  File "/lib/python3.12/site-packages/openai/lib/streaming/_assistants.py", line 102, in until_done
    consume_sync_iterator(self)
  File "/lib/python3.12/site-packages/openai/_utils/_streams.py", line 6, in consume_sync_iterator
    for _ in iterator:
  File "/lib/python3.12/site-packages/openai/lib/streaming/_assistants.py", line 69, in __iter__
    for item in self._iterator:
  File "/lib/python3.12/site-packages/openai/lib/streaming/_assistants.py", line 406, in __stream__
    self._emit_sse_event(event)
  File "demo.py", line 31, in _emit_sse_event
    super()._emit_sse_event(event)
  File "/lib/python3.12/site-packages/openai/lib/streaming/_assistants.py", line 256, in _emit_sse_event
    self.on_event(event)
  File "demo.py", line 22, in on_event
    if tool_call.type == 'code_interpreter':
       ^^^^^^^^^^^^^^
AttributeError: 'dict' object has no attribute 'type'

New environment:

> python -m pip list | grep -E 'openai|pydantic'              
openai                        1.36.0
pydantic                      2.8.2
pydantic_core                 2.20.1

> python --version
Python 3.12.2

> uname -a
Darwin Kernel Version 23.3.0

[RobertCraigie]

Thanks for the report. Unfortunately we're unlikely to be able to prioritise fixing this soon but if you would be willing to put up a PR I'd be happy to review it.