WAFP-Net: Weighted Attention Fusion based Progressive Residual Learning for Depth Map Super-resolution

Abstract

Recently, the development of 3d depth technology brought in many real-world multimedia applications, however, how to tackle real-world degradations in low-resolution (LR) depth maps remains a major challenge, though remarkable progresses have been achieved with DCNN based depth super-resolution (DSR) approaches. Existing DSR models are usually trained and tested on synthetic dataset, which are restrictive and not effective in generalizing to the real-world DSR tasks. In this paper, we aim at alleviating the real-world degradations of different depth sensors in two aspects. First, we classify the generation of LR depth maps into two types: non-linear down-sampling with noise and interval down-sampling, for which different DSR models are learned correspondingly. Second, a novel framework is proposed to handle these two types of LR depth maps in DSR, which consists of four modules: 1) An progressive residual learning module with deep supervision is proposed to learn effective high-frequency components of depth maps in a coarse-to-fine manner. 2) A weighted attention fusion strategy is utilized to intensify the features with abundant high-frequency components in both global and local manners. 3) A multi-stage fusion module is utilized to sufficiently re-exploit the information in the progressive process. 4) A depth refinement module is proposed to improve the depth map by Total Generalized Variation (TGV) regularization and input loss. Extensive experiments on benchmarking datasets demonstrate the superiority of our method over current state-of-the-art DSR methods.