Fast motion estimation for field sequential imaging: Survey and benchmark

• Published in: Image and vision computing 2019-09, Vol.89, p.170-182
• Main Authors: Steiner, Holger, Sommerhoff, Hendrik, Bulczak, David, Jung, Norbert, Lambers, Martin, Kolb, Andreas
• Format: Article
• Language: English
• Subjects: Field sequential imaging; Motion estimation; Optical flow
• ISSN: 0262-8856; 1872-8138
• DOI: 10.1016/j.imavis.2019.07.001

Field sequential (FS) imaging comprises image acquisition systems that capture image channels in temporal sequence in order to provide the final image. A classical application is multispectral imaging. In case of dynamic scenes, the sequential nature of the acquisition imposes motion artifacts, i.e., spatially misaligned images channels. Compensating motion artifacts for this kind of imagery is non-trivial, as common methods for motion estimation rely on the intensity consistency constraint that is violated in FS imaging. This paper surveys approaches to motion compensation in the context of FS imaging. We focus on accuracy in handling intensity inconsistent data and, secondarily, speed, as FS imaging is commonly done in real-time. We introduce a conceptual classification for algorithmic approaches for motion estimation for FS imagery and discuss known and modified approaches to tackle the intensity inconsistencies between adjacent image channels using image transformation and intensity correction methods. As result, we get a set of 379 variants of motion estimation methods applicable to FS data streams. We evaluate these methods using our benchmark database, which comprises data sets from the Middlebury and the MPI Sintel databases, modified to emulate FS imagery, as well as additionally captured multispectral short wave infrared (SWIR) and sRGB image sequences, as well as simulated Time-of-Flight (ToF) image sequences that consist of four channels (called phase images). In order to quantify the motion estimation techniques, we use a ranking scheme similar to Middlebury and combine it with a run-time evaluation. [Display omitted] •An analysis of motion estimation methods applicable to field-sequential imaging.•A benchmark data set including multispectral and time-of-flight imagery.•An in-depth evaluation of motion estimation algorithms and their combinations.