TTS streaming does not work

[meenie]

Confirm this is an issue with the Python library and not an underlying OpenAI API

• This is an issue with the Python library

Describe the bug

When following the documentation on how to use client.audio.speech.create(), the returned response has a method called stream_to_file(file_path) which explains that when used, it should stream the content of the audio file as it's being created. This does not seem to work. I've used a rather large text input that generates a 3.5 minute sound file and the file is only created once the whole request is completed.

To Reproduce

Utilize the following code and replace the text input with a decently large amount of text.

from pathlib import Path
from openai import OpenAI
client = OpenAI()

speech_file_path = Path(__file__).parent / "speech.mp3"
response = client.audio.speech.create(
    model="tts-1",
    voice="alloy",
    input="""
        <Decently large bit of text here>
    """
)

response.stream_to_file(speech_file_path)

Notice that when the script is run that the speech.mp3 file is only ever created after the request is fully completed.

Code snippets

No response

OS

macOS

Python version

Python 3.11.6

Library version

openai v1.2.4

[RobertCraigie]

Thanks for the bug report, we're working on a fix.

[meenie]

Anything I might be able to do to help? I see that there is no mention of streaming in the TTS REST API endpoint, so I'm assuming it doesn't actually support this feature?

[RobertCraigie]

All good! The issue here is that the http client reads the entire response body before returning.

The "stream" terminology here is maybe a little confusing as it's different from the stream argument that you're likely familiar with in the chat.completions.create method as that results in Server-sent Events whereas in this case it's just referring to lazily reading the response body.

[amarbayar]

Once it is fixed, would this allow us to send streamed response from chat.completions.create to the audio.speech.create and get a lazily-reading behavior as text response is streamed in chunks from chat completion?

[antont]

Seems to be fixed in #866

I hacked it in earlier in #724 and results were good, when I just passed the stream=True parameter in .. even for a long (like 30s or more) text, i got to start hearing it in a browser client in about 1s.

@RobertCraigie here seems to say, though, that it only starts when the whole audio is completed on the server? Apparently the generation is quick, then?

[meenie]

@antont oh shoot! I didn't see your issue already about this because I searched TTS and instead of just Speach 🤦🏻. Also, great news about it being possibly fixed! Thanks for verifying :).

[meenie]

Once it is fixed, would this allow us to send streamed response from chat.completions.create to the audio.speech.create and get a lazily-reading behavior as text response is streamed in chunks from chat completion?

@amarbayar, I'm super interested in this! I don't think it will work because the speech endpoint would need to accept a chunk transfer encoding and I don't think it does. How awesome would that be if this was built into the library, though?

[rattrayalex]

@antont out of curiosity, how did you play the streaming audio in the browser? Do you have example code you could share for the benefit of others?

[rattrayalex]

Ah, I'm sorry to say the fix in #866 is being reverted; upon further discussion, we've found a better way that should be available in the coming days. Thank you for your patience.

[EigenSpan]

Ill be honest Im quite new to github as I dont really collab with anyone. Would definitely like to know when this works though. Would love to shave the time off my generated audio. In the meantime while we are waiting for this bugfix is there any other way to stream the response from the tts endpoint?

[antont]

@antont out of curiosity, how did you play the streaming audio in the browser? Do you have example code you could share for the benefit of others?

Well it's very simple: just return audio/mpeg from the http server, and stream the response. The browsers handle that by showing a audio player so didn't need to do anything on the browser side.

I used FastAPI, so the http handler func is:

from fastapi import FastAPI
from fastapi.responses import StreamingResponse
from openai import AsyncStream
from openai.types.chat import ChatCompletionChunk

app = FastAPI()

@app.get("/stream")
async def stream():
    text = "test. "
    text *= 10 #NOTE: I tested with 100 here too to make sure that it streams, i.e. still starts quickly, in about 1s
    
    #was without stream=True param
    #speech_stream: HttpxBinaryResponseContent = await text_to_speech_stream_openai(text)
    speech_stream: AsyncStream[ChatCompletionChunk] = await text_to_speech_stream_openai(text) #is not the actual type

    return StreamingResponse(speech_stream.response.aiter_bytes(), media_type="audio/mpeg")
    
async def text_to_speech_stream_openai(text: str):
    print('Generating audio from text review using Open AI API')
    #without stream=True is: response: HttpxBinaryResponseContent
    stream: AsyncStream[ChatCompletionChunk] = await client.audio.speech.create(
        model="tts-1",
        voice="echo",
        input=text,
        stream=True
    ) #type: ignore
    #print(type(response), dir(response), response)
    print(type(stream), dir(stream), stream)
    return stream

This requires the stream param in speech.create, so either my quick hack PR #724 or the later more proper one from OpenAI, #866 . I guess that one works too even though they reverted it now.

I have this live at https://id-longtask-3q4kbi7oda-uc.a.run.app/stream . First query to the server takes time as it starts up the instance, but it starts playing audio in about 1s in later requests. Don't bomb it too much so I don't need to take it down due to much API usage..

[antont]

Once it is fixed, would this allow us to send streamed response from chat.completions.create to the audio.speech.create and get a lazily-reading behavior as text response is streamed in chunks from chat completion?

Yes proper support for such pipelining within the OpenAI backend would be great.

It might be possible to hack a somewhat working system now

• do the usual streaming chunk based thing to get text
• send it for audio generating intermittently
• start streaming the first audio after a bit of delay, after some text has come in, and the speech stream for that has started
• in the meantime, also read more text from the text response, and start a new stream to get that as audio, to your backend code
• when the first audio stream from openai ends, switch the input to your streaming response to be from the new tts stream
• if the new stream has not started yet, put in some empty audio in your stream or maybe a little pink noise, so it's just a small pause for the listener

If you'd do the splits when a sentence ends it might be bearable, even though might have awkward pauses too. I'm not sure how the timings would go actually, might be even quite fine, maybe even no pauses? Perhaps with a slow speaker :D

[EigenSpan]

I did the split on sentences approach with elevenlabs before openai's tts endpoint came out. It did work but it was quite choppy. I'll just wait for the time being and hope its implemented soon.

Regarding your prior post so did they add the stream parameter to create? Because when I tried to use it in my current version which was updated maybe a week or two ago it said there was no stream parameter.