Abstract

We aim to solve the highly challenging task of generating continuous sign language videos solely from speech segments for the first time. Recent efforts in this space have focused on generating such videos from human-annotated text transcripts without considering other modalities. However, replacing speech with sign language proves to be a practical solution while communicating with people suffering from hearing loss. Therefore, we eliminate the need of using text as input and design techniques that work for more natural, continuous, freely uttered speech covering an extensive vocabulary. Since the current datasets are inadequate for generating sign language directly from speech, we collect and release the first Indian sign language dataset comprising speech-level annotations, text transcripts, and the corresponding sign-language videos. Next, we propose a multi-tasking transformer network trained to generate signer's poses from speech segments. With speech-to-text as an auxiliary task and an additional cross-modal discriminator, our model learns to generate continuous sign pose sequences in an end-to-end manner. Extensive experiments and comparisons with other baselines demonstrate the effectiveness of our approach. We also conduct additional ablation studies to analyze the effect of different modules of our network. A demo video containing several results is attached to the supplementary material.

Abstract

Multi-target tracking of pedestrians is a challenging task due to uncertainty about targets, caused mainly by similarity between pedestrians, occlusion over a relatively long time and a cluttered background. A usual scheme for tackling multi-target tracking is to divide it into two sub-problems: data association and trajectory estimation. A reasonable approach is based on joint optimization of a discrete model for data association and a continuous model for trajectory estimation in a Markov Random Field framework. Nonetheless, usual solutions of the data association problem are based only on location information, while the visual information in the images is ignored. Visual features can be useful for associating detections with true targets more reliably, because the targets usually have discriminative features. In this work, we propose a combination of position and visual feature information in a discrete data association …

Abstract

The annual ACM Innovations in Software Engineering Conference (ISEC) held its 14th edition online during 25–27th February, 2021. Since 2019, the conference has included a PhD symposium. The aim of the symposium is to offer PhD scholars in early stages of their research work a platform to present their work and also to get feedback from the audience. This report from the symposium co-chairs briefly summarises the effort involved in organising the symposium and a brief summary of the papers presented at the symposium.

Abstract

The simplicity of the visual servoing approach makes it an attractive option for tasks dealing with vision-based control of robots in many real-world applications. However, attaining precise alignment for unseen environments pose a challenge to existing visual servoing approaches. While classical approaches assume a perfect world, the recent data-driven approaches face issues when generalizing to novel environments. In this paper, we aim to combine the best of both worlds. We present a deep model predictive visual servoing framework that can achieve precise alignment with optimal trajectories and can generalize to novel environments. Our framework consists of a deep network for optical flow predictions, which are used along with a predictive model to forecast future optical flow. For generating an optimal set of velocities we present a control network that can be trained on the fly without any supervision. Through extensive simulations on photo-realistic indoor settings of the popular Habitat framework, we show significant performance gain due to the proposed formulation vis-a-vis recent state-of-the-art methods. Specifically, we show a faster convergence and an improved performance in trajectory length over recent approaches.

Abstract

— Modern visual-inertial navigation systems (VINS) are faced with a critical challenge in real-world deployment: they need to operate reliably and robustly in highly dynamic environments. Current best solutions merely filter dynamic objects as outliers based on the semantics of the object category. Such an approach does not scale as it requires semantic classifiers to encompass all possibly-moving object classes; this is hard to define, let alone deploy. On the other hand, many realworld environments exhibit strong structural regularities in the form of planes such as walls and ground surfaces, which are also crucially static. We present RP-VIO, a monocular visual-inertial odometry system that leverages the simple geometry of these planes for improved robustness and accuracy in challenging dynamic environments. Since existing datasets have a limited number of dynamic elements, we also present a highly-dynamic, photorealistic synthetic dataset for a more effective evaluation of the capabilities of modern VINS systems. We evaluate our approach on this dataset, and three diverse sequences from standard datasets including two real-world dynamic sequences and show a significant improvement in robustness and accuracy over a state-of-the-art monocular visual-inertial odometry system. We also show in simulation an improvement over a simple dynamic-features masking approach. Our code and dataset are publicly available

Abstract

Embodiments of the present invention provide systems, methods, and computer storage media directed to facilitating corpus-based content generation, in particular, using graph-based multi-sentence compression to generate a final content output. In one embodiment, pre-existing source content is identified and retrieved from a corpus. The source content is then parsed into sentence tokens, mapped and weighted. The sentence tokens are further parsed into word tokens and weighted. The mapped word tokens are then compressed into candidate sentences to be used in a final content. The final content is assembled using ranked candidate sentences, such that the final content is organized to reduce information redundancy and optimize content cohesion.

Abstract

The use of local detectors and descriptors in typical computer vision pipelines work well until variations in viewpoint and appearance change become extreme. Past research in this area has typically focused on one of two approaches to this challenge: the use of projections into spaces more suitable for feature matching under extreme viewpoint changes, and attempting to learn features that are inherently more robust to viewpoint change. In this paper we present a novel framework that combines learning of invariant descriptors through data augmentation and orthographic viewpoint projection. We propose rotation-robust local descriptors, learnt through training data augmentation based on rotation homographies, and a correspondence ensemble technique that combines vanilla feature correspondences with those obtained through rotation-robust features. Using a range of benchmark datasets as well as contributing a new bespoke dataset for this research domain, we evaluate the effectiveness of the proposed approach on key tasks including pose estimation and visual place recognition. Our system outperforms a range of baseline and state-of-the-art techniques, including enabling higher levels of place recognition precision across opposing place viewpoints, and achieves practically-useful performance levels even under extreme viewpoint changes.

Abstract

In this article, we propose a bilevel segmentation framework with metacognitive learning (BS-McL) to detect power lines with an RGB camera mounted on an unmanned aerial vehicle (UAV) platform. The proposed framework consists of two levels based on spectral and spatial techniques. In the first level, spectral classification is carried out using the McL method, which is an evolving online learning neural network architecture. Due to similarities in spectral intensities, few nonpower line pixels are grouped along with power line pixels. The nonpower line pixels are removed by spatial segmentation in the second level. The second level includes morphological operations such as geometric features (shape and density indices), which are applied to detect the power lines. The processing steps of BS-McL are illustrated using a synthetic image of size 9 x 6 pixels. Also, two datasets consisting of 64 images with varying backgrounds, different locations, and dimensions of power lines are used to demonstrate the performance of the proposed BS-McL. The obtained results for BS-McL are compared with five commonly used methods. For both datasets, the efficiency of the BS-McL for power line extraction is better than for the methods used for comparison. Furthermore, the trained knowledge from our experimental set-up (Dataset 1: suburban scene) can be transferred to another dataset that is available publicly (Dataset 2: urban and mountain scenes) if the power line spectral values are in relevance with the distribution in the training dataset. The proposed approach BS-McL is based on online learning with a self-adaptive architecture, which provides improved generalization ability.

Abstract

Recent studies on Question Answering (QA) and Conversational QA (ConvQA) emphasize the role of retrieval: a system first retrieves evidence from a large collection and then extracts answers. This open-retrieval ConvQA setting typically assumes that each question is answerable by a single span of text within a particular passage (a span answer). The supervision signal is thus derived from whether or not the system can recover an exact match of this ground-truth answer span from the retrieved passages. This method is referred to as spanmatch weak supervision. However, information-seeking conversations are challenging for this span-match method since long answers, especially freeform answers, are not necessarily strict spans of any passage. Therefore, we introduce a learned weak supervision approach that can identify a paraphrased span of the known answer in a passage. Our experiments on QuAC and CoQA datasets show that the span-match weak supervisor can only handle conversations with span answers, and has less satisfactory results for freeform answers generated by people. Our method is more flexible as it can handle both span answers and freeform answers. Moreover, our method can be more powerful when combined with the span-match method which shows it is complementary to the span-match method. We also conduct in-depth analyses to show more insights on open-retrieval ConvQA under a weak supervision setting.

Abstract

The paper introduces a novel design for an in-pipe climbing robot with a flexible body, along with the successful simulation of the proposed mechanism. The presented Three-Module pipe climber can traverse a complex network of pipes and bends at various angles. It has bidirectional motion capabilities and is orientation independent. It can also negotiate the ‘Singularity Region’ encountered while turning through the Tjunction. The pipe climber has three modules which are arranged symmetrically apart from each other. These modules mount the tracks which provide the necessary traction for the robot to avoid slippage on the pipe’s surface. The modules, independently, have the ability to compress asymmetrically allowing the robot to bend in any direction. The taper in the front of the design and additional body flexibility due to passive spring compliance aid the motion at turns. The design of the robot is done on SOLIDWORKS and its motion is studied using MSC ADAMS. While simulating the proposed mechanism, differential speed was provided to the tracks for negotiating turns.