Designing Interpretable Recurrent Neural Networks for Video Reconstruction via Deep Unfolding

• Published in: IEEE transactions on image processing 2021, Vol.30, p.4099-4113
• Main Authors: Luong, Huynh Van, Joukovsky, Boris, Deligiannis, Nikos
• Format: Article
• Language: English
• Subjects: Algorithms; Artificial neural networks; Computational modeling; Deep unfolding; Design; Dimensional measurement; Image coding; Image reconstruction; Iterative algorithms; Iterative methods; Minimization; Neural networks; Optimization; Recurrent neural networks; Representations; reweighted <italic xmlns:ali="http://www.niso.org/schemas/ali/1.0/" xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">ℓ ₁-<italic xmlns:ali="http://www.niso.org/schemas/ali/1.0/" xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">ℓ ₁ minimization; sequential frame reconstruction; Signal reconstruction; Task analysis
• ISSN: 1057-7149; 1941-0042
• DOI: 10.1109/TIP.2021.3069296

Deep unfolding methods design deep neural networks as learned variations of optimization algorithms through the unrolling of their iterations. These networks have been shown to achieve faster convergence and higher accuracy than the original optimization methods. In this line of research, this paper presents novel interpretable deep recurrent neural networks (RNNs), designed by the unfolding of iterative algorithms that solve the task of sequential signal reconstruction (in particular, video reconstruction). The proposed networks are designed by accounting that video frames' patches have a sparse representation and the temporal difference between consecutive representations is also sparse. Specifically, we design an interpretable deep RNN (coined reweighted-RNN) by unrolling the iterations of a proximal method that solves a reweighted version of the \ell _{1} - \ell _{1} minimization problem. Due to the underlying minimization model, our reweighted-RNN has a different thresholding function (alias, different activation function) for each hidden unit in each layer. In this way, it has higher network expressivity than existing deep unfolding RNN models. We also present the derivative \ell _{1} - \ell _{1} -RNN model, which is obtained by unfolding a proximal method for the \ell _{1} - \ell _{1} minimization problem. We apply the proposed interpretable RNNs to the task of video frame reconstruction from low-dimensional measurements, that is, sequential video frame reconstruction. The experimental results on various datasets demonstrate that the proposed deep RNNs outperform various RNN models.