Learning Language-Guided Visual Navigation

Abstract

This thesis studies the Embodied AI problem with a focus on Vision-and-Language Navigation (VLN), which an agent is required to follow human natural language instructions and complete certain navigation tasks in unknown environments. The research starts from designing more capable visual-textual grounding models (agents) for decision making, towards building a system for efficient navigation in virtual environments that closely approximate the real world.

We start with enhancing the supervision of learning vision-language correspondence by enriching the benchmark Room-to-Room dataset (R2R) with fine-grained annotations. Our Fine-Grained R2R (FGR2R) matches sub-instructions to viewpoints on sub-trajectories, providing a much stronger and explicit supervision signal compared to the original weakly-supervised data. We propose sub-instruction attention and shifting modules to exploit FGR2R, forcing an agent to solve each sub-task sequentially without distracting by the other less-relevant part of the instruction.

Next, we observe that from both the textual and visual perspectives, the relationships among the scene, its objects, and directional clues are essential for the agent to interpret complex instructions and correctly perceive the environments. In light of this, we introduce a Language and Visual Entity Relationship Graph (Entity-Graph) for modeling the inter-modal relationships between text and vision, and the intra-modal relationships among visual entities. We propose a message passing algorithm for propagating information between language elements and visual entities in the graph, which are then combined to determine the next action to take.

Following the ideas above and inspired by the success of applying transformer-based networks in cross-modal grounding, we create the Recurrent VLN-BERT where the agent's state representation is applied as an input token to the pre-trained transformer, and we directly use the attention weights of images at navigable directions with respect to the state as the action probabilities, thus effectively binding the vision-language grounding and decision making in learning to navigate. Such design allows the agent to adequately benefit from the pre-trained knowledge and largely reduces the computational cost. The method is also generalizable to a wide range of transformer-based architectures.

Then, we address the discrete-to-continuous gap. We propose a candidate waypoint predictor to predict reachable viewpoint options around the agent in continuous environments, allowing the agents to be trained and navigate by image-text grounding as in discrete VLN. Specifically, we transfer the discrete connectivity graphs to the continuous environments for training the waypoint predictor. Then, we leverage the learned navigability to provide accessible directions for the agent designed for discrete VLN to act, which successfully bridges the training and performance gap between agents in the two scenarios.

Lastly, we address the problem of learning visual representations for navigation. Inspired by the behavior that human naturally build semantically and spatially meaningful cognitive maps in their brain during navigation, in this paper, we propose a novel navigational-specific visual representation learning method by contrasting the agent's egocentric views and semantic maps. We apply the visual transformer as the backbone encoder and train the model with data collected from the large-scale Habitat-Matterport3D environments. Our method transfers the compact and rich information from a map, such as objects, structure and transition, to the agent's egocentric representations for navigation and improves the performance. Show more