Abstract

Automatic syllable stress detection methods typically consider syllable-level features as independent. However, as per linguistic studies, there is a dependency among the syllables within a word. In this work, we address this issue by proposing a Post-Net approach using Time-Delay Neural Networks to exploit the syllable dependency in a word for stress detection task. For this, we propose a loss function to incorporate the dependency by ensuring only one stressed syllable in a word. The proposed Post-Net leverages the existing SOTA sequence-independent stress detection models and learns in both supervised and unsupervised settings. We compare the Post-Net with three existing SOTA sequence-independent models and also with sequential model (LSTMs). Experiments conducted on ISLE corpus show the highest relative accuracy improvement of 2.1% and 20.28% with the proposed Post-Net compared to the best sequenceindependent SOTA model in supervised and unsupervised manners, respectively.

Abstract

Automatic syllable stress detection is typically operated at syllable level with stress-related acoustic features. The stress placed on a syllable is influenced not only by its own characteristics but also by its context in the word. However, traditional methods for stress detection overlook the contextual acoustic factors that influence stress placement. By addressing this issue, we study sequential modeling approaches by integrating the syllable dependency for automatic syllable stress detection using a masking strategy. This approach considers a sequence of syllables at the word level and identifies its stress label sequence. We explore various sequential models, such as RNNs, LSTMs, GRUs, and Attention networks. We conduct experiments on the ISLE corpus comprising non-native speakers speaking English. From the experiments, we observe a significant improvement in the performance with all sequential models compared to the state-of-the-art non-sequential baseline (DNN).

Abstract

The task of automatically detecting syllable stress is a key module in computer-assisted language learning systems. There are numerous studies proposed in the literature for automatic syllable stress detection by using different knowledge-based prosodic features. Also, different statistical machine learning and deep learning models are explored for this task using knowledge-based features. However, the acoustic parameters considered to compute knowledge-based features might not always represent the stress phenomena, hence the knowledge based features are not always suitable for generalization and scalability. Recently, the rapidly emerging self-supervised learning based representations are outperforming the existing state-of-the-art knowledge-based features in all speech applications. Also, these representations allow the models to be built in an end-to-end fashion. In this work, we explore the use of self-supervised representations (Wav2Vec-2.0), for syllable stress detection and compare the performance with state-of-the-art knowledge-based features. Further, we use our recently proposed explicit representation learning framework, modeled by jointly optimizing variational autoencoder (VAE) and DNN for stress detection. We analyze the performance of representation learning framework with two different state-of-the-art classifiers, support vector machines (SVM) and simple deep neural network (DNN). We conduct experiments on two non-native English speakers’ datasets from ISLE corpus i.e., German (GER), and Italian (ITA). From the analysis study, it is observed that the classification accuracy for syllable stress detection using self-supervised representations significantly improved by 3.2% and 2.7% over knowledge-based features in GER and ITA, respectively. From the t-SNE plots, it is observed that the representations learned by explicit representation learning framework with VAE show better discrimination among stressed and unstressed syllables compared with representations learned implicitly with simple DNN.

Abstract

Speech systems are sensitive to accent variations. This is a challenge in India, which has numerous languages but few linguistic studies on pronunciation variation. The growing number of L2 English speakers reinforces the need to study accents and L1-L2 interactions. We investigate Indian English (IE) accents and report our observations on regional and shared features. Specifically, we observe phonemic variations and phonotactics in speakers’ native languages and apply this to their English pronunciations. We demonstrate the influence of 18 Indian languages on IE by comparing native language features with IE pronunciations obtained from literature studies and phonetically annotated speech. Hence, we validate Indian language influences on IE by justifying pronunciation rules from the perspective of Indian language phonology. We obtain a comprehensive description of generalised and region-specific IE characteristics, which facilitates accent adaptation of existing speech systems.

Abstract

Automatic syllable stress detection is helpful in assessing L2 learners’ pronunciation. In this work, for stress detection, we propose a representation learning framework by jointly optimizing VAE and DNN. The obtained representations from the proposed VAE plus DNN framework are compared with the implicit representations learned from DNN-based stress detection. Further, we compare the obtained representations from VAE plus DNN with those obtained from autoencoder (AE) plus DNN, and sparse-autoencoder (SAE) plus DNN considering with/without implicit representations from DNN. We perform the experiments on the ISLE corpus consisting of English utterances from German and Italian native speakers. We observe that the detection performance with the learned representations from VAE plus DNN is significantly better than that with the state-of-the-art method without any representation learning with the highest improvement of 2.2%, 5.1%, and 1.4% under matched, combined, and cross scenarios, respectively.

Abstract

Analysis of Indian English (IE) pronunciation variabilities is useful in ASR and TTS modelling for the Indian context. Prior works characterised IE variabilities by reporting qualitative phonetic rules relative to Received Pronunciation (RP). However, such characterisations lack quantitative descriptors and data-driven analysis of diverse IE pronunciations, which could be due to the scarcity of phonetically labelled data. Furthermore, the versatility of IE stems from the influence of a large diversity of the speakers’ mother tongues and demographic region differences. To address these issues, we consider the corpus Indic TIMIT and manually obtain 13, 632 phonetic transcriptions in addition to those parts of the corpus. By performing a data-driven analysis on 15, 974 phonetic transcriptions of 80 speakers from diverse regions of India, we present a new set of phonetic rules and validate them against the existing phonetic rules to identify their relevance. Finally, we test the efficacy of Grapheme-to-Phoneme (G2P) conversion developed based on the obtained rules considering Phoneme Error Rate (PER) as the metric for performance.

Abstract

Second language learners often make grammatical mistakes, which can impact the performance of Spoken Language Understanding (SLU) systems. SLU systems consist of two key components: Automatic Speech Recognition (ASR) and Natural Language Understanding (NLU). This work focuses on the effects of grammatical errors which were synthetically generated by manipulating Parts of Speech (POS) tokens on the NLU module. Our analysis comprises three main aspects. Firstly, we assess the impact of grammatical errors on the overall performance of NLU systems, specifically in the domains of Intent Detection and Slot Filling. Secondly, we investigate NLU performance concerning POS tags. Lastly, we utilize an Attention-based NLU model to evaluate the significance of different POS. This study evaluates NLU models on ATIS and SNIPS datasets for Intent Detection and Slot Filling tasks with introduced grammatical errors, leading to a significant performance drop as large as 4.7% for Intent Detection and 5.1% for Slot Filling. However, training on datasets with synthetic errors helps mitigate this drop, and the models are shown to be sensitive to increasing levels of corrupted tokens. Additionally, attention analysis reveals Proper Nouns’ higher weights in determining intent, while specific POS corruption, particularly Auxiliary Verbs and Nouns, is more likely to cause intent misclassification.

Abstract

The pronunciation quality of second language (L2) learners can be affected by different factors including the fol- lowing seven factors: Intelligibility, Intonation, Phoneme quality, mispronunciation, Mother tongue influence, Correct placement of pause and Correct stress placement. An automatic assessment of these seven factors could be helpful for developing computer- assisted language learning systems. In this work, we assess the quality of all seven factors considering an unsupervised approach using DTW based utterance level alignment distance between expert’s and learner’s speech. Unlike the existing works that consider factor specific heuristic based features, we explore Wav2Vec-2.0 based self-supervised representations as a feature for assessing all the seven factors. The distance is computed using the following three distance metrics: Mean absolute error (MAE), Mean squared error (MSE) and Cosine distance (CD). Experiments are conducted on voisTUTOR corpus containing spoken English speech samples from 16 Indian L2 learners annotated with binary ratings (1 and 0) for all the factors. Using each distance metric, for each learner’s speech, four distance values are computed with a set of four expert samples. In the assessment, to circumvent the need for parallel expert data, we consider two (out of four) expert samples synthesized from the state-of-the-art text-to-speech (TTS) systems. We observe that the performance with the considered distance metric based unsupervised assessment approach is significantly more than that with the baseline for six out of seven factors under all three distance metrics and all four experts samples.

Abstract

The demand for high-quality speech data has been increasing as deep-learning approaches gain popularity in speech applications. Among these, automatic speech recognition (ASR) and text-to-speech (TTS) require large amount of data contain- ing speech and the corresponding text. For these applications, high-quality data is often obtained through manual validation, which ensures matching between speech and text. The manual validation is not scalable as per the demand due to the cost and time involved. In order to cater to the high-quality data demand, validating the data automatically could be useful. In this work, for automatic data validation, a spoken English corpus named IIITH MM2 Speech-Text is created, containing matched and mismatched speech-text pairs under read speech conditions from Indian speakers with different nativities. For the creation, we consider 100 unique stimuli selected from the TIMIT corpus, ensuring phonetic richness, for which a joint entropy maximization is proposed. These stimuli are recorded from 50 speakers, resulting in matched and mismatched sets containing 5000 and 764 utterances with a total duration of 6 hours and 1 hour, respectively. The mismatched set contains speech from the instances where the speakers naturally made spoken errors while reading the reference text. It also contains two stimuli per utterance, one stimulus is the reference text, and the other is manually annotated text that reflects the erroneous speech. Thus, the reference and the annotated text are used for building the models of speech-text mismatch detection and correction, respectively. To the best of our knowledge, no such corpora exist containing both matched and mismatched speech-text. As a preliminary analysis for speech-text mismatch detection, a baseline considering Wav2Vec-2.0 representations and DTW results in the detection F1-score of 0.87

Abstract

Language learning involves the correct acquisition of grammar skills. To facilitate learning with computer-assisted systems, automatic spoken grammatical error detection (SGED) is necessary. This work explores Automatic Speech Recognition (ASR), which decodes text from speech, for SGED. With current advancements in ASR technology, often it can be believed that these systems could capture spoken grammatical errors in the decoded text. However, these systems have an inherent bias from the language model towards the grammatically correct text. We explore the ASR-decoded text from commercially available current state-of-the-art systems considering a text-based GED algorithm and also its word-level confidence score (CS) for SGED. We perform the experiments on the spoken English data collected in-house from 13 subjects speaking 4110 grammatically erroneous and correct sentences. We found the highest relative improvement in SGED with CS is 15.36% compared to that with decoded text plus GED.