Abstract

In light of the recent breakthroughs in automatic machine transla-tion systems, we propose a novel approach that we term as "Face-to-Face Translation". As today’s digital communication becomesincreasingly visual, we argue that there is a need for systems thatcan automatically translate a video of a person speaking in lan-guage A into a target language B with realistic lip synchronization.In this work, we create an automatic pipeline for this problem anddemonstrate its impact in multiple real-world applications. First, webuild a working speech-to-speech translation system by bringing together multiple existing modules from speech and language. Wethen move towards "Face-to-Face Translation" by incorporating anovel visual module, LipGAN for generating realistic talking facesfrom the translated audio. Quantitative evaluation of LipGAN onthe standard LRW test set shows that it significantly outperform sexisting approaches across all standard metrics. We also subjectour Face-to-Face Translation pipeline, to multiple human evalua-tions and show that it can significantly improve the overall user experience for consuming and interacting with multimodal content across languages. Code, models and demo video are made publicly available.

Abstract

No Abstract available

Abstract

While several datasets for autonomous navigation have become available in recent years, they have tended to focus on structured driving environments. This usually corresponds to well-delineated infrastructure such as lanes, a small number of well-defined categories for traffic participants, low variation in object or background appearance and strong adherence to traffic rules. We propose DS, a novel dataset for road scene understanding in unstructured environments where the above assumptions are largely not satisfied. It consists of 10,004 images, finely annotated with 34 classes collected from 182 drive sequences on Indian roads. The label set is expanded in comparison to popular benchmarks such as Cityscapes, to account for new classes. It also reflects label distributions of road scenes significantly different from existing datasets, with most classes displaying greater within-class diversity. Consistent with …

Abstract

Spoof detectors are classifiers that are trained to distinguish spoof fingerprints from bonafide ones. However, state of the art spoof detectors do not generalize well on unseen spoof materials. This study proposes a style transfer based augmentation wrapper that can be used on any existing spoof detector and can dynamically improve the robustness of the spoof detection system on spoof materials for which we have very low data. Our method is an approach for synthesizing new spoof images from a few spoof examples that transfers the style or material properties of the spoof examples to the content of bonafide fingerprints to generate a larger number of examples to train the classifier on. We demonstrate the effectiveness of our approach on materials in the publicly available LivDet 2015 dataset and show that the proposed approach leads to robustness to fingerprint spoofs of the target material.

Abstract

A system and method for capturing Omni-Stereo videos using multi-sensor is disclosed. The system includes left cameras, right cameras and a viewing circle. The method of capturing omni stereo videos using multi-sensor approach includes steps of: capturing images of a scene using left cameras, capturing images of a scene using right cameras, processing each image from the left camera and right camera using a computation method, and obtaining a final omni stereo frame through the computation method.

Abstract

For an autonomous corridor following task where the environment is continuously changing, several forms of environmental noise prevent an automated feature extraction procedure from performing reliably. Moreover, in cases where pre-defined features are absent from the captured data, a well defined control signal for performing the servoing task fails to get produced. In order to overcome these drawbacks, we present in this work, using a convolutional neural network (CNN) to directly estimate the required control signal from an image, encompassing feature extraction and control law computation into one single end-to-end framework. In particular, we study the task of autonomous corridor following using a CNN and present clear advantages in cases where a traditional method used for performing the same task fails to give a reliable outcome. We evaluate the performance of our method on this task on a Wheelchair Platform developed at our institute for this purpose.

Abstract

In this paper, we present AUTORATE, a system that leverages the front camera of a windshield-mounted smartphone to monitor driver’s attention by combining several features. We derive a driver attention rating by fusing spatio-temporal features based on the driver state and behavior such as head pose, eye gaze, eye closure, yawns, use of cellphones, etc. We perform extensive evaluation of AUTORATE on real-world driving data and also data from controlled, static vehicle settings with 30 drivers in a large city. We compare AUTORATE’s automatically-generated rating with the scores given by 5 human annotators. We compute the agreement between AUTORATE’s rating and human annotator rating using kappa coefficient. AUTORATE’s automatically-generated rating has an overall agreement of 0.88 with the ratings provided by 5 human annotators. We also propose soft attention mechanism in AUTORATE which improves AUTORATE’s accuracy by 10%. We use temporal and spatial attention to visualize the key frame and the key action which justify the model’s predicted rating. Further, we observe that personalization in AUTORATE can improve driver specific results by a significant amount

Abstract

: A brain magnetic resonanace imaging (MRI) atlas plays an important role in many neuroimage analysis tasks as it provides an atlas with a standard coordinate system which is needed for spatial normalization of a brain MRI. Ideally, this atlas should be as near to the average brain of the population being studied as possible.

Abstract

Optical coherence tomographic (OCT) images provide valuable information for understanding the changes occurring in the retina due to glaucoma, specifically, related to the retinal nerve fiber layer and the optic nerve head. In this paper, we propose a deep learning approach using Capsule network for glaucoma classification, which directly operates on 3D OCT volumes. The network is trained only on labelled volumes and does not attempt any region/structure segmentation. The proposed network was assessed on 50 volumes and found to achieve 0.97 for the area under the ROC curve (AUC). This is considerably higher than the existing approaches which are majorly based on machine learning or rely on segmentation of the required structures from OCT. Our network also outperforms 3D convolutional neural networks despite the fewer network parameters and fewer epochs needed for training.

Abstract

Localizing natural language phrases in images is a challenging problem that requires joint understanding of both the textual and visual modalities. In the unsupervised setting, lack of supervisory signals exacerbate this difficulty. In this paper, we propose a novel framework for unsupervised visual grounding which uses concept learning as a proxy task to obtain self-supervision. The intuition behind this idea is to encourage the model to localize to regions which can explain some semantic property in the data, in our case, the property being the presence of a concept in a set of images. We present thorough quantitative and qualitative experiments to demonstrate the efficacy of our approach and show a 5.6% improvement over the current state of the art on Visual Genome dataset, a 5.8% improvement on the ReferItGame dataset and comparable to state-of-art performance on the Flickr30k dataset.