Abstract

Convolutional neural networks (CNN) applied to Mel spectrograms now have a dominant presence in the landscape of speaker recognition systems. Correspondingly, it is also important to evaluate their robustness to adversarial attacks that remains not thoroughly explored for end-to-end trained CNNs for speaker recognition. Our work addresses this gap and investigates variations of the iterative Fast Gradient Sign Method (FGSM) to perform adversarial attacks. We observe that a vanilla iterative FGSM can flip the identity of each speaker sample to that of every other speaker in the LibriSpeech dataset. Furthermore, we propose adversarial attacks specific to Mel spectrogram features by (a) limiting the number of pixels attacked, (b) restricting changes to specific frequency bands, (c) restricting changes to particular time duration, and (d) using a substitute model to craft the adversarial sample. Using thorough qualitative and quantitative results, we demonstrate the fragility and non-intuitive nature of the current CNN-based speaker recognition systems, where the predicted speaker identities can be flipped without any perceptible changes in the audio. The samples are available at "https://advdemo.github.io/speech/"

Abstract

Humans have a natural ability to perform semantic associations with the surrounding objects in the environment. This allows them to create a mental map of the environment, allowing them to navigate on-demand when given linguistic instructions. A natural goal in Vision Language Navigation (VLN) research is to impart autonomous agents with similar capabilities. Recent works take a step towards this goal by creating a semantic spatial map representation of the environment without any labeled data. However, their representations are limited for practical applicability as they do not distinguish between different instances of the same object. In this work, we address this limitation by integrating instance-level information into spatial map representation using a community detection algorithm and utilizing word ontology learned by large language models (LLMs) to perform open-set semantic associations in the mapping representation. The resulting map representation improves the navigation performance by two-fold (233%) on realistic language commands with instance-specific descriptions compared to the baseline. We validate the practicality and effectiveness of our approach through extensive qualitative and quantitative experiments.

Abstract

The prose storyboard language is a formal language for de- scribing movies shot by shot, where each shot is described with a unique sentence. The language uses a simple syntax and limited vocabulary borrowed from working practices in traditional movie-making, and is intended to be readable both by machines and humans. The language is designed to serve as a high-level user interface for intelligent cinematog- raphy and editing systems

Abstract

We investigate Referring Image Segmentation (RIS), which outputs a segmentation map corresponding to the natural language description. Addressing RIS efficiently requires considering the interactions happening across visual and linguistic modalities and the interactions within each modality. Existing methods are limited because they either compute different forms of interactions sequentially (leading to error propagation) or ignore intramodal interactions. We address this limitation by performing all three interactions simultaneously through a Synchronous Multi-Modal Fusion Module (SFM). Moreover, to produce refined segmentation masks, we propose a novel Hierarchical Cross-Modal Aggregation Module (HCAM), where linguistic features facilitate the exchange of contextual information across the visual hierarchy. We present thorough ablation studies and validate our approach’s performance on four benchmark datasets, showing considerable performance gains over the existing state-of-the-art (SOTA) methods.

Abstract

Despite existing works of Audio-Visual Saliency Prediction (AVSP) models claiming to achieve promising results by fusing audio modality over visual-only models, these models fail to leverage audio information. In this paper, we investigate the relevance of audio cues in conjunction with the visual ones and conduct extensive analysis by employing well-established audio modules and fusion techniques from diverse correlated audio-visual tasks. Our analysis on ten diverse saliency datasets suggests that none of the methods worked for incorporating audio. Furthermore, we bring to light, why AVSP models show a gain in performance over visual-only models, though the audio branch is agnostic at inference. Our work questions the role of audio in current deep AVSP models and motivates the community to a clear avenue for reconsideration of the complex architectures by demonstrating that simpler alternatives work equally well.

Abstract

No Results Found

Abstract

We present a method to control the emotional prosody of Text to Speech (TTS) systems by us- ing phoneme-level intermediate features (pitch, energy, and duration) as levers. As a key idea, we propose Differential Scaling (DS) to dis- entangle features relating to affective prosody from those arising due to acoustics conditions and speaker identity. With thorough exper- imental studies, we show that the proposed method improves over the prior art in accu- rately emulating the desired emotions while retaining the naturalness of speech. We ex- tend the traditional evaluation of using indi- vidual sentences for a more complete evalua- tion of HCI systems. We present a novel ex- perimental setup by replacing an actor with a TTS system in offline and live conversa- tions. The emotion to be rendered is either predicted or manually assigned. The results show that the proposed method is strongly pre- ferred over the state-of-the-art TTS system and adds the much-coveted “human touch” in ma- chine dialogue. Audio samples for our exper- iments and the code are available at: https: //emtts.github.io/tts-demo/

Abstract

Kathakali is one of the major forms of Classical Indian Dance. The dance form is distinguished by the elaborately colourful makeup, costumes and face masks. In this work, we present (a) a framework to analyze the facial expressions of the actors and (b) novel visualization techniques for the same. Due to extensive makeup, costumes and masks, the general face analysis techniques fail on Kathakali videos. We present a dataset with manually annotated Kathakali sequences for four downstream tasks, i.e. face detection, background subtraction, landmark detection and face segmentation. We rely on transfer learning and fine-tune deep learning models and present qualitative and quantitative results for these tasks. Finally, we present a novel application of style-transfer of Kathakali video onto a cartoonized face. The comprehensive framework presented in the paper paves the way for better understanding, analysis, pedagogy and visualization of Kathakali videos.

Abstract

Domain Generalization (DG) requires a model to learn a hypothesis from multiple distributions that generalizes to an unseen distri- bution. Recent explorations show that, for neural networks, the choice of hyper-parameters and model architecture significantly affects DG performance, and making the right choice is non-trivial. In this paper, we show evidence suggesting that the models that perform better at DG, might be implicitly learning a low dimen- sional representation in the feature space. Furthermore, we take forward this idea and employ explicit feature learning to improve DG. To this end, we propose a DG specific supervised contrastive loss. We show how this performance improvement correlates to reduced dimensionality of the representation. Our work establishes new state-of-the-art on five different DG benchmarks, compared against over two dozen existing approaches in DomainBed. We show how this performance improvement correlates to reduced dimensionality of the representation

Abstract

Kathakali is one of the major forms of Classical Indian Dance. The dance form is distinguished by the elaborately colourful makeup, costumes and face masks. In this work, we present (a) a framework to analyze the facial expressions of the actors and (b) novel visualization techniques for the same. Due to extensive makeup, costumes and masks, the general face analysis techniques fail on Kathakali videos. We present a dataset with manually annotated Kathakali sequences for four downstream tasks, i.e. face detection, background subtraction, landmark detection and face segmentation. We rely on transfer learning and fine-tune deep learning models and present qualitative and quantitative results for these tasks. Finally, we present a novel application of style-transfer of Kathakali video onto a cartoonized face. The comprehensive framework presented in the paper paves the way for better understanding, analysis, pedagogy and visualization of Kathakali videos. CCS Concepts: Applied computing --> Performing arts; Human-centered computing --> Visualization