Abstract

Video highlights are a selection of the most interesting parts of a video. The problem of highlight de-tection has been explored for video domains like egocentric,sports, movies, and surveillance videos. Existing methods are limited to finding visually important parts of the video but does not necessarily learn semantics. Moreover, the available benchmark datasets contain audio muted, single activity,short videos, which lack any context apart from a few keyframes that can be used to understand them. In this work,we explore highlight detection in the TV series domain, which features complex interactions with the surroundings. The existing methods would fare poorly in capturing the video semantics in such videos. To incorporate the importance of dialogues/audio, we propose using the descriptions of shots of the video as cues to learning visual importance.Note that while the audio information is used to determine visual importance during training, the highlight detection still works using only the visual information from videos.We use publicly available text ranking algorithms to rank the descriptions. The ranking scores are used to train a visual pair wise shot ranking model (VPSR) to find the highlights of the video. The results are reported on TV series videos of the Video Set dataset and a season of Buffy the Vampire SlayerTV series.

Abstract

wo-view stereo problem is a well researchedproblem in 3D computer vision. Algorithms proposedin the past have focussed on rectified stereo imageswhere the epipolar lines are parallel to the horizon-tal axis.The general problem of computing stereocorrespondences for unrectified images without anyknowledge of calibration parameters is an importantproblem but unexplored as yet. Our idea in this paperis to predict depth maps from two unrectified stereoimages using a modified Flownet architecture. Since,datasets for depth map reconstruction for unrectifiedstereo images for deep learning do not exist, we havecreated a dataset of turn table sequences of 3D modelsfrom Google 3D warehouse. Following the conceptsof Attention modelling, we implement an architecturefor combining correlations computed at multipleresolutions using a simple element-wise multiplicationof the correlations to aid the architecture to resolvecorrespondences for textureless and repeated texturedsurfaces. Our experiments show both qualitaitve andquantitative improvements of depth maps over theoriginal Flownet architecture.

Abstract

We look at the problem of developing a compact and accurate model for gesture recognition from videos in a deep-learning framework. Towards this we propose a joint 3DCNN-LSTM model that is end-to-end trainable and is shown to be better suited to capture the dynamic information in actions. The solution achieves close to state-of-the-art accuracy on the ChaLearn dataset, with only half the model size. We also explore ways to derive a much more compact representation in a knowledge distillation framework followed by model compression. The final model is less than 1 MB in size, which is less than one hundredth of our initial model, with a drop of 7% in accuracy, and is suitable for real-time gesture recognition on mobile devices.

Abstract

In computer vision, many active illumination techniques em-ploy Projector-Camera systems to extract useful information from the scenes. Known illumination patterns are projected onto the scene and their deformations in the captured images are then analyzed. We observe that the local frequencies in the captured pattern for the mirror-like surfaces is different from the projected pattern. This property allows us to design a custom Projector-Camera system to segment mirror-like surfaces by analyzing the local frequencies in the captured images. The system projects a sinusoidal pattern and capture the images from projector’s point of view. We present segmentation results for the scenes including multiple reflections and inter-reflections from the mirror-like sur-faces. The method can further be used in the separation of direct and global components for the mirror-like surfaces by illuminating the non-mirror-like objects separately. We show how our method is also useful for accurate estimation of shape of the non-mirror-like regions in the presence of mirror-like regions in a scene.

Abstract

Fine-grained classification is an extremely challenging problem in computer vision, compounded by subtle differences in shape, pose, illumination and appearance. While convolutional neural networks have become the versatile jack-of-all-trades tool in modern computer vision, approaches for fine-grained recognition still rely on localization of keypoints and parts to learn discriminative features for recognition. In order to achieve this, most approaches use a localization module and subsequently learn classifiers for the inferred locations,thus necessitating large amounts of manual annotations for bounding boxes and key points. In order to tackle this problem,we aim to leverage the (taxonomic and/or semantic) relationships present among fine-grained classes. The ontology tree is a free source of labels that can be used as auxiliary tasks to train a multi-task loss. Additional tasks can act as regularizers, and increase the generalization capabilities of the network. Multiple tasks try to take the network in diverging directions, and the network has to reach a common minimum by adapting and learning features common to all tasks in its shared layers.We train a multi-task network using auxiliary tasks extracted from taxonomical or semantic hierarchies, using a novel method to update task-wise learning rates to ensure that the related tasks aid and unrelated tasks does not hamper performance on the primary task. Experiments on the popular CUB-200-2011 dataset show that employing super-classes in an end-to-end model improves performance, compared to methods employing additional expensive annotations such as key points and bounding boxes and/or using multi-stage pipelines.

Abstract

We present a practical solution for generating 360◦stereo panoramic videos using a single camera. Current approaches either use a moving camera that captures multiple images of a scene, which are then stitched together to form the final panorama, or use multiple cameras that are synchronized. A moving camera limits the solution to static scenes, while multi-camera solutions require dedicated calibrated setups. Our approach improves upon the existing solutions in two significant ways: It solves the problem using a single camera, thus minimizing the calibration problem and providing us the ability to convert any digital camera into a panoramic stereo capture device. It captures all the light rays required for stereo panoramas in a single frame using a compact custom designed mirror, thus making the design practical to manufacture and easier to use. We analyze several properties of the design as well as present panoramic stereo and depth estimation results.

Abstract

Transgender face recognition is gaining increasing attention in the face recognition community because ofits potential in real life applications. Despite extensive progress in traditional face recognition domain, it is very challenging to recognize faces under transgender setting.The gender transformation results in significant face variations, both in shape and texture gradually over time. This introduces additional complexities to existing face recognition algorithms to achieve a reliable performance. In this paper, we present a novel framework that incorporates appearance factor and a transformation factor caused due to Hormone Replacement Therapy (HRT) for recognition.To this extent, we employ the Hidden Factor Analysis(HFA) to jointly model a face under therapy as a linear combination of appearance and transformation factors.This is based on the intuition that the appearance factor captures the features that are unaffected by the therapy and transformation factor captures the feature changes due to therapy. Extensive experiments carried out on publicly available HRT transgender face database shows the efficacy of the proposed scheme with a recognition accuracy of82.36%

Abstract

The problem of labeling the edges present in a single color image as convex, concave, and occluding entities is one of the fundamental problems in computer vision. It has been shown that this information can contribute to segmentation, reconstruction and recognition problems. Recently, it has been shown that this classification is not straightforward even using RGBD data. This makes us wonder whether this apparent simple cue has more information than a depth map? In this paper, we propose a novel algorithm using random forest for classifying edges into convex, concave and occluding entities. We release a data set with more than 500 RGBD images with pixel-wise ground labels. Our method produces promising results and achieves an F-score of 0.84 on the data set.

Abstract

In this work, we propose an online handwriting solution, where the data is captured with the help of depth sensors. Users may write in the air and our method recognizes it in real time using the proposed feature representation. Our method uses an efficient fingertip tracking approach and reduces the necessity of pen-up/pen-down switching. We validate our method on two depth sensors, Kinect and Leap Motion Controller. On a dataset collected from 20 users, we achieve a recognition accuracy of 97.59% for character recognition. We also demonstrate how this system can be extended for lexicon recognition with reliable performance. We have also prepared a dataset containing 1,560 characters and 400 words with the intention of providing common benchmark for handwritten character recognition using depth sensors and related research.

Abstract

Recently, exemplar based approaches have been successfully applied for face detection in the wild. Contrary to traditional approaches that model face variations from a large and diverse set of training examples, exemplar-based approaches use a collection of discriminatively trained exemplars for detection. In this paradigm, each exemplar casts a vote using retrieval framework and generalized Hough voting, to locate the faces in the target image. The advantage of this approach is that by having a large database that covers all possible variations, faces in challenging conditions can be detected without having to learn explicit models for different variations. Current schemes, however, make an assumption of independence between the visual words, ignoring their relations in the process. They also ignore the spatial consistency of the visual words. Consequently, every exemplar word contributes equally during voting regardless of its location. In this paper, we propose a novel approach that incorporates higher order information in the voting process. We discover visual phrases that contain semantically related visual words and exploit them for detection along with the visual words. For spatial consistency, we estimate the spatial distribution of visual words and phrases from the entire database and then weigh their occurrence in exemplars. This ensures that a visual word or a phrase in an exemplar makes a major contribution only if it occurs at its semantic location, thereby suppressing the noise significantly. We perform extensive experiments on standard FDDB, AFW and G-album datasets and show significant improvement over previous exemplar …