Adaptively Meshed Video Stabilization

• Published in: IEEE transactions on circuits and systems for video technology 2021-09, Vol.31 (9), p.3504-3517
• Main Authors: Zhao, Minda, Ling, Qiang
• Format: Article
• Language: English
• Subjects: Cameras; Feature extraction; feature trajectories; Finite element method; Frames (data processing); Mesh generation; Motion simulation; Optimization; Parallax; Stabilization; Three-dimensional displays; Trajectory; Transformations; Transmission line matrix methods; triangle mesh; Two dimensional displays; Video; Video stabilization
• ISSN: 1051-8215; 1558-2205
• DOI: 10.1109/TCSVT.2020.3040753

Video stabilization is essential for improving the visual quality of shaky videos. Current video stabilization methods usually take feature trajectories in the background to estimate one global transformation matrix or several transformation matrices based on a fixed mesh, and warp shaky frames into their stabilized views. However, these methods may not model the shaky camera motion well in complicated scenes, such as scenes containing large foreground objects or strong parallax, and may result in notable visual artifacts in the stabilized videos. To resolve the above issues, this paper proposes an adaptively meshed method to stabilize a shaky video based on all of its feature trajectories and an adaptive blocking strategy. More specifically, we first extract the feature trajectories of the shaky video and then generate a triangle mesh according to the distribution of the feature trajectories in each frame. Then, the transformations between shaky frames and their stabilized views over all triangular grids of the mesh are calculated to stabilize the shaky video. Since more feature trajectories can usually be extracted from all of the regions, including both the background and foreground regions, a finer mesh will be obtained and provided for camera motion estimation and frame warping. We estimate the mesh-based transformations of each frame by solving a two-stage optimization problem. Moreover, foreground and background feature trajectories are no longer distinguished and both contribute to the estimation of the camera motion in the proposed optimization problem, yielding better estimation performance than previous works, particularly in challenging videos with large foreground objects or strong parallax. To further enhance the robustness of our method, we propose two adaptive weighting mechanisms to improve its spatial and temporal adaptability. Experimental results demonstrate the effectiveness of our method in producing visually pleasing stabilization effects in various challenging videos.