Abstract

In this paper, we introduce a novel approach to address the task of synthesizing speech from silent videos of any in-the-wild speaker solely based on lip movements. The traditional approach of directly generating speech from lip videos faces the challenge of not being able to learn a robust language model from speech alone, resulting in unsatisfactory outcomes. To overcome this issue, we propose incorporating noisy text supervision using a state-of-the-art lip-to-text network that instills language information into our model. The noisy text is generated using a pre-trained lip-to-text model, enabling our approach to work without text annotations during inference. We design a visual text-to-speech network that utilizes the visual stream to generate accurate speech, which is in-sync with the silent input video. We perform extensive experiments and ablation studies, demonstrating our approach's superiority over the current state-of-the-art methods on various benchmark datasets. Further, we demonstrate an essential practical application of our method in assistive technology by generating speech for an ALS patient who has lost the voice but can make mouth movements. Our demo video, code, and additional details can be found at http://cvit.iiit.ac.in/research/projects/cvit-projects/ms-l2s-itw.

Abstract

Stuttering is a speech disorder that affects speech fluency and rhythm, with millions worldwide experiencing it. Early diagnosis and treatment can significantly improve speech fluency and the quality of life for individuals who stutter. Automatic detection of stuttering events can help diagnose, monitor, and develop effective interventions. Therefore, this paper aims to propose a feature space-based classifier for detecting stuttering events in speech. To achieve this, we have investigated Zero Time Windowing Cepstral Coefficients (ZTWCC) as a feature set for stutter detection using classifiers such as SVM, LSTM, and Bidirectional LSTM. We compared the performance of ZTWCC with the standard handcrafted features, such as MFCC, CQCC, and SFFCC, on the SEP- 28K dataset with and without including phase information. The results in both cases indicate that ZTWCC is giving a higher F-1 score than baseline MFCC features.

Abstract

Despite the progress seen in classification methods, current approaches for handling videos with distribution shifts in source and target domains remain source-dependent as they require access to the source data during the adaptation stage. In this paper, we present a self-training based source-free video domain adaptation approach to address this challenge by bridging the gap between the source and the target domains. We use the source pre-trained model to generate pseudo-labels for the target domain samples, which are inevitably noisy. Thus, we treat the problem of source-free video domain adaptation as learning from noisy labels and argue that the samples with correct pseudo-labels can help us in adaptation. To this end, we leverage the crossentropy loss as an indicator of the correctness of the pseudo-labels and use the resulting small-loss samples from the target domain for fine-tuning the model. We further enhance the adaptation performance by implementing a teacher-student framework, in which the teacher, which is updated gradually, produces reliable pseudo-labels. Meanwhile, the student undergoes fine-tuning on the target domain videos using these generated pseudo-labels to improve its performance. Extensive experimental evaluations show that our methods, termed as CleanAdapt, CleanAdapt + TS, achieve state-of-the-art results, outperforming the existing approaches on various open datasets. Our source code is publicly available at https://avijit9.github.io/CleanAdapt.

Abstract

Language learning involves the correct acquisition of grammar skills. To facilitate learning with computer-assisted systems, automatic spoken grammatical error detection (SGED) is necessary. This work explores Automatic Speech Recognition (ASR), which decodes text from speech, for SGED. With current advancements in ASR technology, often it can be believed that these systems could capture spoken grammatical errors in the decoded text. However, these systems have an inherent bias from the language model towards the grammatically correct text. We explore the ASR-decoded text from commercially available current state-of-the-art systems considering a text-based GED algorithm and also its word-level confidence score (CS) for SGED. We perform the experiments on the spoken English data collected in-house from 13 subjects speaking 4110 grammatically erroneous and correct sentences. We found the highest relative improvement in SGED with CS is 15.36% compared to that with decoded text plus GED.

Abstract

Large language models (LLMs) and generative AI have emerged as the most important areas in the field of natural language processing (NLP). LLMs are considered to be a key component in several NLP tasks, such as summarization, question-answering, sentiment classification, and translation. Newer LLMs, such as Chat- GPT, BLOOMZ, and several such variants, are known to train on multilingual training data and hence are expected to process and generate text in multiple languages. Considering the widespread use of LLMs, evaluating their efficacy in multilingual settings is imperative. In this work, we evaluate the newest generative models (ChatGPT, mT0, and BLOOMZ) in the context of Indic languages. Specifically, we consider natural language generation (NLG) applications such as summarization and questionanswering in monolingual and cross-lingual settings. We observe that current generative models have limited capability for generating text in Indic languages in a zero-shot setting. In contrast, generative models perform consistently better on manual quality-based evaluation in Indic languages and English language generation. Considering limited generation performance, we argue that these LLMs are not intended to use in zero-shot fashion in downstream applications.

Abstract

This paper introduces a novel method for determining the best room to place an object in, for embodied scene rearrangement. While state-of-the-art approaches rely on large language models (LLMs) or reinforcement learned (RL) policies for this task, our approach, CLIPGraphs, efficiently combines commonsense domain knowledge, data-driven methods, and recent advances in multimodal learning. Specifically, it (a) encodes a knowledge graph of prior human preferences about the room location of different objects in home environments, (b) incorporates vision-language features to support multimodal queries based on images or text, and (c) uses a graph network to learn object-room affinities based on embeddings of the prior knowledge and the vision-language features. We demonstrate that our approach provides better estimates of the most appropriate location of objects from a benchmark set of object categories in comparison with state-of-the-art baselines

Abstract

Social media provides a supportive and anonymous environment for discussing mental health issues, including depression. Existing research on identifying the cause of depression focuses primarily on improving classifier models, while neglecting the importance of learning better data representations. To address this gap, we introduce an architecture that enhances the identification of the cause of depression by learning improved data representations. Our work enables a deeper interpretation of the cause of depression in social media contexts, emphasizing the significance of effective rep- resentation learning for this task. Our work can act as a foundation for self-help applications in the field of mental health.

Abstract

With increasing connectivity and sophisticated software, modern vehicles are able to leverage different kinds of services provided by the environment. One such service recommended by the Automotive Edge Computing Consortium (AECC) is downloading high-definition map data by vehicles. This high volume of data can be provided to the vehicles when moving by pre-allocating resources on edge server nodes or roadside units if the routes are known apriori. However, this is not a realistic assumption to make in general. Therefore, in this work, we propose a two-stage optimization framework for efficient data delivery to connected vehicles via edge nodes while considering dynamic route changes. We have evaluated the efficiency of this proposed approach (considering a real-world dataset) with respect to (a) offline optimization strategies considering fixed routes and (b) a greedy approach considering route changes. Our proposed approach works considerably better than the existing approaches in the context of dynamic route changes.

Abstract

Unmanned Aerial Vehicles (UAVs), particularly low-cost UAVs, have become increasingly important due to their wide range of applications and ease of use. However, with the rapid growth of the UAV market, the rising security concerns pose a greater risk. One such primary concern is location spoofing attacks which can compromise UAV's navigation system, making it crucial to analyze various location-based attacks. In this paper, we identify 16 such GPS spoofing scenarios based on environmental conditions, attack type, and spoof signal propagation path. We evaluate these scenarios based on various GPS parameters like Horizontal Dilution Of Precision (HDOP), Vertical Dilution Of Precision (VDOP), GPS satellite count in view, and avg signal-to-noise power density (CN0). We then analyze the variations in GPS parameters for various such attack scenarios. Further, we analyze the impact of distance on average CN0 and the effect of satellite count on effective spoofable distance. We also discuss several critical insights which are empirically observed during our experimental trials. Our experiments revealed that the natural conditions within indoor and outdoor scenarios can vary considerably, and effective spoofable distance can be up to 100 meters when the satellite count is less than 10.

Abstract

The high rate of groundwater usage has resulted in a rapid decline in groundwater levels, which has necessitated its monitoring. This paper focuses on estimating the groundwater level inside borewells remotely and in real-time using the Internet of Things (IoT). For this, a solution is proposed which is based on string tension and does not require any electrical components to be lowered into the borewell. The solution provides a completely automated, real-time, reliable, and IoT-enabled alternative to existing methods for borewell water monitoring. The proposed approach is compared with an ultrasonic sensor-based approach in a controlled environment as well as in a tank. Additionally, four nodes were deployed in a small educational campus in the Indian city of Hyderabad to ascertain its usability and reliability in practical situations. The observations from the data collected over a month show that the proposed low-cost solution is reliable and has a good performance in the field. Index Terms—Borewell, Real-time, IoT, String tension, Water-level.