Video communication with mobile devices is a more and more relevant service for current and next-generation wireless networks. Therefore, a huge research effort has been devoted to design coding and transmission systems to get the best video quality for a given amount of wireless resources. This is particularly challenging when network and channel characteristics change quickly over time.
One central idea of this project is that it is possible to arbitrarily reduce the latency of any video codec and transmission system by displaying at the receiver a prediction of the currently acquired image before it is actually received. Once finally received, the image is used to predict the next ones, thus limiting the transmitter/receiver drift. Using this framework we can, in principle, reduce the latency of the video codec by an arbitrary amount, so that the network latency can be partially or even totally compensated for. Nevertheless, this latency reduction comes at the cost of the displayed images’ fidelity, since they are based on temporal extrapolation. Thus, the latency-quality trade-off is central to this project.
- Low-latency conventional codecs remove backward temporal prediction (low-delay profiles) or remove temporal prediction at all (JPEG-XS). However, they cannot compensate for network latency
- Traditional temporal extrapolation methods, based on optical flow and motion compensation, have been outperformed by recent CNN-based techniques, which however cannot perfectly take into account the effect of conventional transmission systems, since the latter are not modeled as differentiable operators
- Linear Video Coding has been proposed for wireless video transmission as an effective alternative to scalable coding; it does not suffer from digital cliff in case of channel degradation but introduces large encoding delays because of the use of GOP-wise transform
Our project builds on the idea of using temporal extrapolation to arbitrarily reduce end-to-end latency; in addition, LVC allows for end-to-end learning-based optimization
Recent publications:
GLASS-TO-GLASS DELAY REDUCTION: ENCODING RATE REDUCTION VS. VIDEO FRAME EXTRAPOLATION Proceedings Article Forthcoming
In: IEEE Workshop on Multimedia Signal Processing, IEEE, Forthcoming.