Our team is currently doing research and developing prototypes about the use of Forward Error Correction for video transmissions.
Forward Error Correction (FEC) [1] is a well-known technique for controlling errors in data transmission. In IP networks, errors actually translate into the loss of an entire UDP or TCP datagram, which is why FEC is also referred to as Forward Erasure Correction. The goal is then to be able to-recreate a lost packet, based on other transmitted packets and redundant information included in specific FEC packets.
Why are there packet losses?
Packet losses [2] are common in telecommunications due to the nature of transmissions: for example, a wireless network can be subject to radio frequency interference. Other causes of packet drops are temporary congestion occurring in routers.
In the real world, it is common and usually acceptable to observe around a 1‒2 % loss rate for many kinds of internet connectivity (xDSL, LTE, 5G, Wifi ...).
Why are packet losses a problem?
In most situations of internet usage, packet losses are handled by the TCP protocol: the receiver acknowledges the received packets, and stops until the sender retransmits the lost packets.
However, this technique is not applicable in the field of real-time communications, because it is not acceptable to stop the streaming until the retransmitted packets arrive.
This is the reason why real-time audio and video streams are preferably transmitted over UDP protocol. Though application-level retransmissions can be implemented efficiently over UDP, they are often not suitable because of round-trip delays between the sender and receiver.
Packet losses are well handled for audio streams thanks to Packet Loss Concealment (PLC), which makes them almost un-noticeable to the end users as long as the loss rate remains below 5%.
However, they are far more complicated to handle for video data. Indeed, the nature of video coding means that a single loss within a video frame makes the whole frame un-decodable, and the next frames too, until a special frame called a "key-frame" is received. This is a consequence of the differential coding that is used for video compression.
How are packet losses handled in real-time video transmissions?
When a video receiver is observing a packet loss, it may notify the remote sender of the loss of information thanks to special RTCP packets (PLI - packet loss indication [3]), so that the sender can immediately generate a key-frame to let the receiver continue the rendering of the video sequence.
This technique, which has been implemented in linphone for many years, has the drawback of freezing the video display until the said key-frame is received.
For a medium quality video stream and with a 1% loss rate, the freeze events will not happen frequently.
However, for a high quality (HD) video stream, the amount of data is so high that even a 0.5% loss rate will result in video freezing almost every second, causing a non-smooth, jerky effect for the end-user.
Why can FEC help?
FEC offers a solution to compensate the loss rates typically observed over the Internet. At the expense of a relatively low overhead and small latency, the redundancy information transmitted by the sender lets the receiver re-create the missing packets, as long as the loss rate remains compatible with the amount of redundancy sent by the encoder.
As a result, video can be played smoothly, without any observable effect for the end-user.
How will this be handled in Linphone?
There are currently various FEC standard techniques and approaches documented at IETF.
Our team has selected the Flexible Forward Error Correction scheme (flexfec) documented in RFC8627 [4], because it combines low-complexity redundancy coding (XOR based) with a high-level of flexibility for the FEC parameters, which is key to adjusting the redundancy overhead according to the observed loss rate and network bandwidth.
The results look very promising!
We will be soon very pleased to announce a future release of linphone and linphone-sdk, incorporating an FEC algorithm for video transmission!
Annex :
[1] https://en.wikipedia.org/wiki/Error_correction_code#Forward_error_correc...
[2] https://en.wikipedia.org/wiki/Packet_loss
