updated the stt streaming code to track and calculate message latency #6
Conversation
- minor change to stream_audio_file.py to track audio cursor - large refactor to print_transcript.py to calculate interim result and EOT message latency for both Nova and Flux - updated README to reflect changes
|
@jjmaldonis you are correct in your description of how hard it is to actually define EOT latency (and also latency in general). for the most accurate downstream benchmarking script assuming no ground truth data, the best approach would likely be to use SileroVAD and measure the delta between the last SileroVAD activity and the Flux EOT message, but even that relies on SileroVAD accuracy as a dependency. the current approach is actually fairly favorable to Flux re: "the amount of time it took the EOT to trigger after Deepgram finished processing the most recent non-EOT message." in related customer documentation we should we should call this out explicitly. actual EOT benchmarking shows a p50 EOT latency of closer to 320ms, compared to the above p50 of 116ms. Flux processes audio in 80ms increments, regularly decodes every 240ms, and triggers additional decodes at 80ms increments when EOT thresholds have been reached. I think this setup is generally on par/useful for customers and is probably not worth getting into the SileroVAD EOT stuff, though one thing that stands out is looking at 'Message Latency' across Flux vs Nova: knowing how Nova operates via endpointing vs Flux via regular Updates and EOT triggered decodes, I know this to be accurate. that said, it still feels a bit misleading in that the customer experience with Flux vs. Nova will be that Flux is generally faster. because this benchmarking is relative to |
We've needed to update our STT latency scripts for a while. This code is meant to do that. It tracks two types of STT latency: interim result latency and EOT latency.
interim_result=truemessages or (for Flux) for Update messages. The code performs the typicalaudio cursor - transcript cursorcalculation.How this script measures EOT latency:
When an EOT message is received (speech_final, is_final, UtteranceEnd, EndOfTurn, or EagerEndOfTurn), the calculation is simple: find the prior interim_result / Update message and subtract the
receivedtimes. The result is the amount of wall clock time it took to receive an EOT signal after the prior interim result was received; said another way, it's the amount of time it took the EOT to trigger after Deepgram finished processing the most recent non-EOT message.There are better ways to calculate EOT latency, but all of them (to my knowledge) require ground truth timestamps and careful labeling of the audio. Since we don't have that information, I believe the current calculation is a reasonable approximation.
Below is a snippet of the output from the
print_transcript.pyscript for a Flux transcript:And below are the same sections for a Nova transcript:
You'll notice that the format of the messages/transcript is similar between both Flux and Nova, and you'll also notice that the summarized latency data at the end is representative of the significant improvements that Flux provides, particularly for EOT latency.