A fast super-resolution technique for multi-view video plus depth transmission

Abstract

The rise in the availability of 3D-enabled devices has made the capture, transmission and display of 3D content a popular research area within the scientific community. Factors such as bandwidth and video quality need to be taken into consideration so as to provide the end-user with an immersive quality experience. Auto-stereoscopic displays allow for an immersive viewing experience by displaying 3D scenes from multiple camera views. These displays generally make use of depth-image-based rendering techniques, which in turn rely heavily on depth maps. Depth maps describe a given scene using greyscale pixel values to convey depth, hence each camera view has a corresponding depth map. The depth and texture views can be encoded using Multi-View High Efficiency Video Coding, were the texture and depth videos are encoded and transmitted separately. Although the encoder makes use of a variety of techniques to compress the content efficiently, the bandwidth requirements increase linearly with the number of transmitted views. In order to reduce bandwidth requirements, the approach used in this dissertation is to down-sample the texture views, and apply super-resolution during the up-sampling process at the receiver. The super-resolution method chosen is a dictionary-based method, and the dictionary is trained using a convolutional neural network. Two dictionaries were used for testing to study the performance with different parameters. Using a dictionary means that it is loaded once on the receiver end, which reduces the restoration time of the frame. The super-resolution algorithm is implemented on a Graphics Processing Unit (GPU), and is scaled to work on an increasing number of GPUs by splitting the image into tiles. Once the tiles are super-resolved, they are then stitched back together to form a super-resolved frame. The visual quality of the algorithm was evaluated using a combination of subjective and objective tests. The results indicate that the super-resolution algorithm performs very well for the test sequences chosen, however it struggles to recover finer details, such as car number plates. The GPU implementation was successful in reducing the execution time using one of the dictionaries, and the scaling of the algorithm proved to increase the number of frames computed per second for both dictionaries tested.