Prototype: Real-Time AI-Powered call transcription

Prototype: Real-Time AI-Powered call transcription

As part of our work on real-time communications, we have implemented a prototype for real-time transcription of a voice stream into text (Speech-to-Text), based on open-source automatic speech recognition models.

Our goal was to explore the possibility of transcribing speech in real time, directly within the application, without relying on an external service.

Objectives: improving real-time transcription quality while keeping data local

Launched in 2025, this project quickly raised several technical challenges.

The first challenge was to achieve transcription that was both fast and reliable. In the context of a real-time audio call, text needs to appear on screen as the conversation unfolds, almost simultaneously with the other participant’s speech. We therefore needed to find the right balance between transcription accuracy and processing speed, while keeping display latency to a minimum.

Protecting the confidentiality of conversations was another major concern. Our objective was to perform all processing locally, without transmitting potentially sensitive data to third-party services. This required selecting a model that was powerful enough to ensure good transcription quality, including in noisy environments, while keeping resource consumption under control.

Finally, choosing an open-source solution was essential. Since all the software we develop is open source, we also wanted to rely on open-source technologies.

To identify the most suitable AI technology, we evaluated several automatic speech recognition models. We selected two models: Whisper and Vosk, both open-source solutions capable of running locally on the user’s device. Whisper stands out for its support for numerous languages and its robustness across a wide range of audio conditions. Vosk offers lightweight models suited to offline use on devices with limited resources.

Our prototype makes it possible to test different options and evolve the transcription engine independently of the rest of the application, thanks to a modular API.

From audio to usable text

Real-time transcription comes with several challenges. Audio is processed in segments, with their size and overlap determined by balancing processing speed against the amount of available context. These overlaps can lead to repetitions or errors. The partial results generated by the engine then need to be integrated into the text displayed on screen.

We therefore focused on managing these segments and improving intermediate transcription results. Performance can be assessed according to three criteria: transcription latency, required resources and error rate.

Potential applications for Linphone

Real-time transcription opens up several possibilities for a softphone such as Linphone:

• Live call captioning, displaying text as the conversation unfolds.
• Improved accessibility for deaf and hard-of-hearing people.
• Easier conversation tracking, particularly when listening conditions are difficult.
• Preparing meeting summaries from transcripts, or providing a foundation for automatic translation features.

Local processing is particularly valuable when it comes to confidentiality. When both audio and its transcription remain on the device, there is no need to transmit this data to a cloud service.

A prototype, not yet a Linphone feature

This work is still at the prototype stage. Further testing is needed to optimise performance and determine the conditions under which integration would be relevant.

It represents a first step towards exploring transcription features within an open-source communication solution, with data processed locally.