Naslov
Dense Semantic Forecasting in Video by Joint Regression of Features and Feature Motion

Autori
Saric, Josip ; Vrazic, Sacha ; Segvic, Sinisa

Izvornik
IEEE Transactions on Neural Networks and Learning Systems (2162-237X) Early access (2021); 1-13

Vrsta, podvrsta i kategorija rada
Radovi u časopisima, članak, znanstveni

Ključne riječi
Computer vision, deep learning, dense semantic forecasting, future prediction.

Sažetak
Dense semantic forecasting anticipates future events in video by inferring pixel-level semantics of an unobserved future image. We present a novel approach that is applicable to various single-frame architectures and tasks. Our approach consists of two modules. Feature-to-motion (F2M) module forecasts a dense deformation field that warps past features into their future positions. Feature-to-feature (F2F) module regresses the future features directly and is therefore able to account for emergent scenery. The compound F2MF model decouples the effects of motion from the effects of novelty in a task-agnostic manner. We aim to apply F2MF forecasting to the most subsampled and the most abstract representation of a desired single-frame model. Our design takes advantage of deformable convolutions and spatial correlation coefficients across neighbouring time instants. We perform experiments on three dense prediction tasks: semantic segmentation, instance-level segmentation, and panoptic segmentation. The results reveal state-of-the-art forecasting accuracy across three dense prediction tasks.

Izvorni jezik
Engleski

Znanstvena područja
Računarstvo

POVEZANOST RADA