Deep sequence learning for video anticipation: from discrete and deterministic to continuous and stochastic

Abstract

Video anticipation is the task of predicting one/multiple future representation(s) given limited, partial observation. This is a challenging task due to the fact that given limited observation, the future representation can be highly ambiguous. Based on the nature of the task, video anticipation can be considered from two viewpoints: the level of details and the level of determinism in the predicted future. In this research, we start from anticipating a coarse representation of a deterministic future and then move towards predicting continuous and fine-grained future representations of a stochastic process. The example of the former is video action anticipation in which we are interested in predicting one action label given a partially observed video and the example of the latter is forecasting multiple diverse continuations of human motion given partially observed one. In particular, in this thesis, we make several contributions to the literature of video anticipation. Firstly, we introduce a general action anticipation framework in which, given very limited observation, the goal is to predict the action label as early as possible. This task is highly critical in scenarios where one needs to react before the action is finalized. This is, for instance, the case of automated driving, where a car needs to, e.g., avoid hitting pedestrians and respect traffic lights. Our work builds on the following observation: a good anticipation model requires (i) a good video representation that is discriminative enough even in presence of partial observation; and (ii) a learning paradigm that not only encourages correct predictions as early as possible, but also accounts for the fact that the future is highly ambiguous. On publicly available action recognition datasets, our proposed method is able to predict markedly accurate action categories given very limited observation, e.g., less than 2\% of the videos of UCF-101, outperforming the state of the art methods by a large margin. Secondly, we proposed an action anticipation in driving scenarios. Since there was no anticipation-specific dataset covering generic driving scenarios, as part of our second contribution, we introduced a large-scale video anticipation dataset, covering 5 generic driving scenarios, with a total of 25 distinct action classes. It contains videos acquired in various driving conditions, weathers, daytimes and environments, complemented with a common and realistic set of sensor measurements. . We then focus on the continuous future prediction problem on tasks that are stochastic in nature; given one observation, multiple plausible futures are likely. In particular, we target the problem of human motion prediction, i.e., the task of predicting future3D human poses given a sequence of observed ones. To this end, in our third contribution, we propose a novel diverse human motion prediction framework based on variational autoenecoders (VAEs). In this approach, we particularly propose a novel stochastic conditioning scheme that is well-suited for scenarios where we are dealing with a deterministic datasets, with strong conditioning signals, and expressive decoders. Through extensive experiments, we show that our approach performs much better than existing approaches and standard practices in training a conditional VAE. Finally, in the fourth contribution, we propose a conditional VAE framework that solves two main issues of a standard conditional VAE: (i) conditioning and sampling the latent variables are two independent processes, and (ii) the prior distribution is set to be unconditional in practice, however, it should be conditioned on the conditioning signal as elaborated in the evidence lower bound of the data likelihood. In our proposed approach, we address both of these issues that leads to substantial improvement in the quality of generated samples. Show more