Faster streaming support

[ameenba]

Have you tried building the spectrogram and encoder output in smaller chunks and appending? I think the spectrogram should generate fairly easily with minimal noise depending on the size of the chunk, and the encoder output can also be appended with sufficiently large chunks.

So the encoder instead takes in Bx80xN as its input and outputs Bx(N/2)x(embedding size), if you wanted to send in 1s of audio into the tiny.en model, for example, 1x80x100->1x50x384. This should result in much faster processing for short clips (when the audio clip is <30s), and allows real time streaming without much wasted computation (like having to calculate a full x1500 encoding for each chunk of audio).

Some noise may be introduced at various size of chunks (spectrogram chunk size can be independent from encoder chunk size), and some overlap of spectrogram input/encoder output may help further to reduce that noise. This allows us for better scheduled deployment where the decoder, encoder, and spectrogram can run on different threads at the same time to produce our transcription.

Choosing when to decode will be another challenge as you don't want to decode if a full word is not complete in the encoding, but there are definitely solutions around that as well.

[ggerganov]

Oh my god! I just tested that and it seems to work o.O
I reduced the audio context by half and the performance doubled. jfk.wav transcribed correctly!

Very interesting...
I have to go now, but I think this is a big breakthrough. Need to double check.

[ggerganov]

Here is diff if someone wants to play with it:

git diff
diff --git a/whisper.cpp b/whisper.cpp
index 7078863..df47bff 100644
--- a/whisper.cpp
+++ b/whisper.cpp
@@ -1053,6 +1053,7 @@ static bool whisper_model_load(const std::string & fname, whisper_context & wctx
             return false;
         }
     }
+    model.e_pe->ne[1] /= 2;
 
     fin.close();
 
@@ -1076,7 +1077,7 @@ static bool whisper_encode(
     const auto & mel_inp = wctx.mel;
     const auto & hparams = model.hparams;
 
-    const int n_ctx   = hparams.n_audio_ctx;
+    const int n_ctx   = hparams.n_audio_ctx/2;
     const int n_state = hparams.n_audio_state;
     const int n_head  = hparams.n_audio_head;
     const int n_layer = hparams.n_audio_layer;
@@ -1474,7 +1475,7 @@ static bool whisper_decode(
     const int n_layer = hparams.n_text_layer;
 
     const int N = n_tokens;
-    const int M = hparams.n_audio_ctx;
+    const int M = hparams.n_audio_ctx/2;
 
     struct ggml_init_params params = {
             .mem_size   = wctx.buf_compute.size(),

[gitslav]

@ggerganov amazing work on this project 👍
Another demo idea: Typing comments while doing an asciinema cast is so 2021...

[trholding]

#10 (comment)

Noise removal such as with rrnoise https://jmvalin.ca/demo/rnnoise/ also increases accuracy of whisper even if sped up a bit. Also had a good experience with "audio companding / compression" in sox along with band pass. Maybe simple audio filtering could help in accuracy when run 2x...

Here is an example: https://github.com/audo-ai/magic-mic/tree/main/src-native

[ggerganov]

@trholding

Thanks for the ideas - these are very interesting to me.

Initially, I was very hopeful for the idea of increasing the tempo combined with partially evaluating the encoder as described in the original comment above by @ameenba. It seemed that it would be possible for example to process in real-time 4 second chunks by speeding the tempo to get 2 second chunks and running the Encoder with a context of N/2=100 (i.e. 2 sec) instead of the original N/2=1500 (30 sec) which would in theory give about x15 speed-up for the Encoder. This can easily run in real-time using the base model on a Raspberry Pi 4 for example.

Unfortunately, my experiments show that when I "trim" the Encoder so much, the Decoding fails miserably - usually repeating the first word over and over again. I tried to resolve that by "stitching" sequential Encoder outputs hoping to stabilize the Decoder, but it didn't work out for some reason.

In fact, using anything less than N/2 < 512 (i.e 10 sec) audio context seems to be unstable and at this point the idea for tempo increase becomes not so lucrative in the context of real-time streaming.

Sure, you can use it to process long audio (i.e. > 30 sec) faster, but for this use case you can simply use a 3rd-party software to pre-process the audio (e.g. sox, ffmpeg, etc.) before feeding it to Whisper, so it no longer makes sense to support it on the whisper.cpp level.

VAD and rnnoise could be useful, but same argument as above is valid here as well. In general, unless it is some very simple and robust algorithm for pre-processing that can be easily implemented in C/C++ without any extra dependencies, then I don't have interest in adding it to the project.

Still, the original idea above for audio context reduction with N/2 = 512 (without tempo increase) so far seems to be useful. For example, it is now possible to run the tiny model at 4 second time steps in real-time on RPi4, which I think is getting close to something that can have actual real-world applications.

Next, I want to add asynchronous interface to the C API which will make it easier to create streaming applications for iOS and WebAssembly. I expect with the current performance to be able to create some nice real-time transcription demos on these platforms.

[ameenba]

@ggerganov thanks for the investigation!

I also found 5-10s to be close to the limit, I haven't fully investigated from my end in python but curious on what you've tried also, my next steps were:

• see how much overlap (of mel or encoder) would reduce noise
• maybe theres a smarter way instead of overlapping? I'm not an audio expert
• allow KV audio cache size to be dynamic, and investigate effects of changing encoding on each decode cycle? I.e. if the cache size is changing on each new encoding, so first set of decodes have a 5s cache, 1x250x384, and the next set has 10s due to appending, do we need to start over with decoding? Will an offset prevent this? Can we treat this as a totally new decode set with maybe some overlap to prevent words being cut off?

I haven't added the -su option at the moment, I'm trying to investigate this streaming approach in isolation before adding loss from other sources. Although I'm sure like you said, there's a lot of preprocessing that could be done to the input, such as speeding up the speech or cutting noise from outside the vocal range?

I'd like to clarify the current streaming approach too: if you were to encode in 5s increments for a 15s audio sample, you would effectively generate 3 encodings with 5, 10, and 15s of relevant audio each (expanding window), with a padded spectrogram, and so each chunk would cost the same computation as a 30s encoding, and for the final translation, only the last encoding is relevant. in this example, the encoder is 3x less efficient for streaming, while the decoding is roughly the same.
So the first big jump possible would the dynamically sized encoder allows for a 5s, 10s, and 15s encoding (afterwards you pad the encoding output directly before decoding, rather than padding the spectrogram), which will roughly translate to the same computation as a single 30s encoding. And of course the dream is to be able to run this as three independent 5s encodings, effectively reducing our computation by 50%, or however long the audio is divided by 30s.

[ggerganov]

@ameenba

• Haven't performed much tests regarding the overlap - I experimented mostly with 0 ms overlap of audio between sequential chunks, because it is most obvious how to append the encoder outputs one after the other.
• I tested only with fixed-size KV cache, padding with 0s for the chunks that are not yet available. Always starting the decoding from the beginning. I'll probably test your dynamic cache-size suggestion when I have some time

Also, one important note about the encoder - for each new chunk I use the corresponding position embeddings, by extracting a sub-tensor from the encoder.positional_embedding tensor:

https://github.com/ggerganov/whisper.cpp/blob/1ac2665488f228e18f02c3692f2aa8205635459f/whisper.cpp#L1148-L1151

Not sure if this is correct, but I think it makes sense because this is the way to tell the transformer how the audio is ordered in time.

Another thing that I have in mind is that it is obvious that appending the encoder outputs in this way is not equivalent to processing the audio all together. The question is - does this produce at least an approximation of the original result, or is there a better strategy to mix them?

I haven't added the -su option at the moment, I'm trying to investigate this streaming approach in isolation before adding loss from other sources.

The -su option might become useful only if the encoding starts working with ~1 second chunks. So I wouldn't worry about it for now.

I'd like to clarify the current streaming approach too: if you were to encode in 5s increments for a 15s audio sample, you would effectively generate 3 encodings with 5, 10, and 15s of relevant audio each (expanding window), with a padded spectrogram, and so each chunk would cost the same computation as a 30s encoding, and for the final translation, only the last encoding is relevant. in this example, the encoder is 3x less efficient for streaming, while the decoding is roughly the same.

Correct. To clarify, the decoding time is proportional to the number of decoded tokens.

So the first big jump possible would the dynamically sized encoder allows for a 5s, 10s, and 15s encoding (afterwards you pad the encoding output directly before decoding, rather than padding the spectrogram), which will roughly translate to the same computation as a single 30s encoding. And of course the dream is to be able to run this as three independent 5s encodings, effectively reducing our computation by 50%, or however long the audio is divided by 30s.

Yes, I agree.
Alternative real-time "adaptive" strategy that I had in mind is like this:

• Run tiny.en at each 5s second chunk to get rough transcription fast - i.e. for t=[0,5], t=[5, 10], t=[10, 15]
• Then, while processing t=[15, 20], run a bigger model in parallel on t=[0, 15] and improve the transcription retrospectively

[nyadla-sys]

##I guess below logic may work better for stream application

pcmf32.resize(n_samples_new);

  SDL_DequeueAudio(g_dev_id_in, pcmf32.data() , n_samples_new*sizeof(float));

#if 0
static int total_samples = 0;
total_samples += n_samples_new;
if (total_samples <= 480000) {
memcpy(&pcmf32_to_keep[total_samples - n_samples_new], pcmf32.data(), n_samples_new * sizeof(float));
} else {
memmove(&pcmf32_to_keep[0], &pcmf32_to_keep[n_samples_new], (n_samples_30s - n_samples_new) * sizeof(float));
memcpy(&pcmf32_to_keep[n_samples_30s - n_samples_new], pcmf32.data(), n_samples_new * sizeof(float));
}
#else

   static int n_buffer_samples = 0;
   if( (n_buffer_samples+n_samples_new) > n_samples_30s)
      pcmf32_to_keep.resize(n_buffer_samples+n_samples_new);

   for (int i = 0; i < n_samples_new; i++) {
      pcmf32_to_keep[n_buffer_samples+i] = pcmf32[i] ;
   }

   n_buffer_samples = n_buffer_samples+n_samples_new;

   if(n_buffer_samples >= n_samples_30s){
      pcmf32 = std::vector<float>(pcmf32_to_keep.end() - n_samples_30s, pcmf32_to_keep.end());
      pcmf32_to_keep.resize(n_samples_30s);
      pcmf32_to_keep = pcmf32;
      n_buffer_samples = n_samples_30s;
   }

#endif

  //Generate spectrograms
  const auto processor_count = std::thread::hardware_concurrency();
  if (!log_mel_spectrogram(pcmf32_to_keep.data(), pcmf32_to_keep.size(), WHISPER_SAMPLE_RATE, WHISPER_N_FFT, WHISPER_HOP_LENGTH, WHISPER_N_MEL, processor_count,filters, mel)) {
    fprintf(stderr, "%s: failed to compute mel spectrogram\n", __func__);
    return -1;
  }

//Run inference here

[dragonman225]

Hi guys, I've been tinkering with whisper.cpp for a while to explore how well it works for streaming/real-time applications. So far I'm able to get not bad transcription quality with reasonable CPU usage. I would like to share my progress so far.

First, a demo video of the result:

out.mp4

The CPU usage on an M1 MacBook Air is around 100% to 200% (1 core = 100%), leaving much of the resources for other applications that people may want to run alongside, for example, a video call. Note that I wrapped whisper.cpp in an Electron app with node-addon-api, so the actual %CPU used by whisper should be even lower.

Here are the tricks I'm using:

• I use the smallest tiny.en model.
• Whisper runs at 400ms step.
• 4 threads seem to be the sweet spot for 8-core M1. Having more or less results in worse performance.
• Setting audio_ctx to 750 (half of the default), as explored eariler, do speed up transcription by 2x. In my case, it reflects through about 2x lower CPU usage.
• Use simple VAD like in the "stream" example to decide when a sentence ends. My parameters for the VAD function are: 3 seconds of audio, detecting silence in the last 450ms, vad_thold 0.3, freq_thold 200.
  • If no sentence end is detected for too long (14 seconds), for example, when a person speaks so fast that there's no long enough gap, force an end and listen for a new sentence.
• Each sentence is preceded by 0.2 seconds of audio of the previous sentence, in case the cut is in the middle of a word.

Here is my code if you want to check out: https://github.com/tableos/mina/blob/main/native/stt_whisper.cc#L109

At this moment I'm satisfied with the quality-performance balance of this setup, in terms of streaming/real-time application, and I'm most wondering if there're other low-hanging fruit to try to further improve quality or performance. (For example, maybe, using a more accurate VAD such Silero VAD.)