Vision and Language Navigation Using Minimal Voice Instructions

Abstract

The proposed system aims to design an algorithm
that can be used to navigate any 3-D mapped environment, using
the Matterport 3D Simulator by giving only minimal voice
instructions. During the training phase, the nodes of a selected
environment are traversed sequentially in the Simulator and an
object recognition algorithm is applied on the panorama at each
node. This helps in identifying and tagging the objects in the
vicinity of each viewpoint. For the testing phase, a natural
language instruction, specifying the goal location is taken as
input. The goal location is identified from among the various
viewpoints in the 3D environment by matching it to the tags
generated in the testing phase. A shortest path algorithm is
employed to navigate from the starting location to the goal
location. The proposed system focuses on the implementation of
the algorithm which combines natural language processing and
computer vision and can be employed by agents for indoor
navigation.