A hybrid smartphone-cloud based rendering algorithm for natural video scenes on mobile device

Abstract

Free-Viewpoint television is one of the most complex applications that requires a substantial amount of processing power. This application requires a set of depth and camera views that will be used to create a virtual viewpoint for user consumption. The Free- Viewpoint algorithm can be broken down into a number of discrete tasks. These tasks can then be distributed between the cloud and mobile device in an optimal manner to benefit from the capabilities of both the cloud and the mobile device. While the cloud can easily scale in terms of processing capacity, the mobile device is limited to its on-board processing capability and battery capacity. Although mobile device technology is constantly evolving with ever increasing processing capabilities being introduced with each new generation of devices, consideration needs to be taken to a wide variety of Mobile devices already in the market whose capabilities vary significantly. These constraints all need to be taken into account in the mobile side implementation. One further important consideration is dependant on the quality of the network that provides the connectivity between the cloud and the mobile devices. Mobile and WIFI networks are also evolving at a fast pace offering ever increasing speeds and low latency, however wireless networks are a shared media which varies significantly in performance depending on the radio conditions and loading. After analysing the Free-Viewpoint algorithm and its stages it was decided to implement a system where the cloud executes the warping of the left and right frames, the generation of a blended image. This image is then compressed using a lossy compression algorithm and is sent to the mobile device over a TCP socket on request. Once the mobile device receives the view it performs in-painting to fill the holes and displays the image. The implemented hybrid system generated the blended view, using OpenMP parallelisation libraries, in 0.7891 seconds on the cloud side, but further improvement could be done. While on mobile device two in-painting techniques were implemented and tested in order to assess the effectiveness of the algorithms and their performance in terms of speed of execution. Both algorithms performed in-painting effectively however there was significant difference in time it took to process a frame. The natively implemented in-painting technique took on average 1.6 seconds on the latest mobile device while the in-built in-painting function provided with the OpenCV libraries took on average 0.089 second, on the same device.