Abstract

A system and a method for recommending background music that corresponds to an extracted text from a book based on emotion and a topic that is relevant to the extracted text using machine learning models provided. The method includes,(i) determining, using a first trained machine learning model, the emotion from the extracted text that corresponds to the paragraph of the book,(ii) assigning, using a word similarity technique, a similarity score for

Abstract

Personalized Interactive Music Systems (PIMSs) are emerging as promising devices for enhancing physical activity and exercise outcomes. By leveraging real-time data and adaptive technologies, PIMSs align musical features, such as tempo and genre, with users’ physical activity patterns, including frequency and intensity, enhancing their overall experience.

Abstract

Background Despite showing remarkable performance on tasks across modalities, deep neural networks remain opaque, with limited insight on how they internally organize information. A learned hierarchy is suggested in vision (Zhou et al., 2015) and language (Tenney et al., 2019) models in deep learning. Bregman (1990) describes the process of auditory scene analysis (ASA), where the auditory system abstracts information based on perceptual properties such as pitch and timbre, among others, in order to construct auditory streams that are then integrated to form higher order percepts like musical genre. This implies that the representation of audio goes through a series of hierarchical abstractions in the process. While hierarchical representations are indicated in human auditory processing (Kell et al., 2018), the same has not been thoroughly demonstrated in deep learning models for audio. Aims We study how convolutional neural networks (CNNs) represent the hierarchy of audio tasks, hypothesizing that representations at earlier layers of a CNN perform better at lower level tasks, and those at later layers perform better at higher level tasks. Methods On the basis of Bregman’s ASA model, we choose tasks that are arranged in a hierarchy from the domains of speech and music. We select three hierarchical musical tasks - note identification, instrument classification, and genre classification, representative of low-, mid-, and high-level tasks. We use class-balanced subsets of NSynth (Engel et al., 2017) with 1800 instances across 12 classes, Medley-solos-DB (Lostanlen et al., 2019) with 965 instances across 7 classes , and GTZAN (Tzanetakis et al., 2001) with 1000 instances across 10 classes, for each of the respective tasks. We inspect three models - VGGish (Hershey et al., 2017), CLAP (Elizalde et al., 2023), and MobileNetV3 (Schmid et al., 2023). We use k-Nearest Neighbour classifiers with class-balanced five-fold cross-validation to assess the model's accuracy on each task with intermediate representations extracted at six equally spaced convolutional blocks and the first fully-connected layer. We repeat this for three hierarchically related speech tasks - consonant classification, keyword recognition, and speaker count estimation, using similarly class-balanced subsets of PCVC (Malekzadeh et al., 2020) with 1794 instances across 23 classes, Speech Commands (Warden, 2018) with 1750 instances across 35 classes, LibriCount (Stöter et al., 2018) with 1100 instances across 11 classes respectively. Results Our results show that the CNN’s early layers learned low-level tasks and later layers learned high-level tasks even without explicit training. On CLAP, for note identification, the first layer’s accuracy (47%) is greater than later layers (<30%). For instrument classification, we see peak accuracy at the last layer (97%), but a remarkable change occurs at the fourth layer (71%) compared to the third layer (55%). Genre classification accuracy at the last layer (75%) is better than earlier layers (<66%). These trends hold across all models. We see similar trends for the speech tasks. Discussion The results support our hypothesis, providing strong evidence that CNNs implicitly learn a hierarchy in sound. This mirrors the human brain’s hierarchical encoding and deserves further investigation if other deep learning architectures and training methodologies also encode similar hierarchies. We also see how low-level tasks are implicitly learnt.

Abstract

Background Music Emotion Recognition (MER) systems primarily rely on audio features (Wang et al., 2021), with recent approaches incorporating lyrics to analyze sentiment and structure for improved accuracy (Agrawal et al., 2021). Musical content, particularly melody, has been found to have a stronger capacity to convey perceived emotional expression than lyrics, although lyrics tend to enhance negative emotions more easily than positive emotions (Ali & Peynircioğlu, 2006). This variability raises important questions: Which component carries the greatest emotional information? Do listeners rely on music, vocals, or lyrics when assessing emotional content? Aim Here, we systematically analyse the contribution of music, vocals, and lyrics to perception of emotion in music. Methods For this study, we used the DEAM (Soleymani et al., 2013) and PMEmo (Zhang et al., 2018) datasets, comprising a total of 2,700 songs. The DEAM dataset includes 1,802 audio items annotated for both dynamic and static emotion using continuous valence and arousal ratings. Participants utilised a two-dimensional interface based on the Self‑Assessment Manikins (SAM) to continuously rate the emotional content at a 2‑Hz sampling rate. The PMEmo dataset contains 794 popular music choruses sourced from international music charts (Billboard Hot 100, iTunes Top 100, and UK Top 40) and was annotated using a similar interface to capture continuous dynamic ratings for valence and arousal at the same sampling rate. Songs were source-separated using Ultimate Vocal Remover (Takahashi et al., 2017) to isolate music and vocals, while lyrics were transcribed using OpenAI’s Whisper model (Radford et al., 2022) and verified manually. Deep learning models trained on diverse song and speech datasets predicted VA values for musical and vocal components, while lyrics were analyzed using models trained on general and lyrically annotated texts (Çano, 2017). Spearman's correlation was calculated between predicted and human-annotated (HA) VA values. Additionally, quadrant-based (Q1: positive V, high A; Q2: negative V, high A; Q3: negative V, low A; Q4: positive V, low A) concurrency analyses evaluated prediction alignment with human-identified emotional quadrants. Results The musical component exhibited the highest correlation with HA ratings (valence: r = 0.70, arousal: r = 0.74), followed by vocals (valence: r = 0.54, arousal: r = 0.65; all p < .001), while lyrics contributed the least (valence: r = 0.11, arousal: r = 0.01). In terms of overall quadrant concurrency, the musical component showed the highest concurrency (60.41%) followed by vocals (48.84%) and then lyrics (31.45%). Specifically, the highest quadrant concurrency was observed in Q3 (musical: 90.53%, vocal: 93.53%, lyrics: 66.16%) followed by Q1 for musical (64.51%) and vocals (39.10%), and Q4 for lyrics (42.50%). Discussion Our findings emphasize the dominant role of the musical component in shaping perceived emotional expression in Western tonal music, aligning with prior work highlighting melody’s emotional salience (Ali & Peynircioğlu, 2006). Furthermore, for music signifying negative valence and low arousal (Q3), there is a higher degree of congruence between the components in conveying sadness and its related emotions. These results suggest that MER systems could prioritize musical and vocal components over lyrical content (Wang et al., 2021; Agrawal et al., 2021), as they appear to primarily drive the perception of emotional expression. References Agrawal, Y., Shanker, R. G. R., & Alluri, V. (2021). Transformer-based approach towards music emotion recognition from lyrics. In V. U. Rao & R. Kailash (Eds.), Advances in information retrieval (pp. 167–175). Springer. https://doi.org/10.1007/978-3-030-72240-1_12 Ali, S. O., & Peynircioğlu, Z. F. (2006). Songs and emotions: Are lyrics and melodies equal partners? Psychology of Music, 34(4), 511–534. https://doi.org/10.1177/0305735606067168 Çano, E. (2017). MoodyLyrics: A sentiment-annotated lyrics dataset. In Proceedings of the 8th International Workshop on Semantic Evaluation (SemEval 2017) (pp. 118–124). Association for Computational Linguistics. https://doi.org/10.18653/v1/S17-1017 Radford, A., Kim, J. W., Xu, T., Brockman, G., McLeavey, C., & Sutskever, I. (2022). Robust speech recognition via large-scale weak supervision. arXiv. https://doi.org/10.48550/arXiv.2212.04356 Soleymani, M., Caro, M., Schmidt, E., Sha, C.-Y., & Yang, Y.-H. (2013). 1000 songs for emotional analysis of music. In P. Herrera, E. Gómez, & T. Kaneko (Eds.), Proceedings of the 2nd ACM International Workshop on Music Information Retrieval with User-Centered and Multimodal Strategies (pp. 1–6). Association for Computing Machinery. Takahashi, N., Huang, P.-S., & Lee, P. (2017). Multi-scale multi-band DenseNets for audio source separation. arXiv. https://doi.org/10.48550/arXiv.1706.09588 Wang, S., Xu, C., Ding, A. S., & Tang, Z. (2021). A Novel Emotion-Aware Hybrid Music Recommendation Method Using Deep Neural Network. Electronics, 10(15), 1769. https://doi.org/10.3390/electronics10151769 Zhang, K., Zhang, H., Li, S., Yang, C., & Sun, L. (2018). The PMEmo dataset for music emotion recognition. In Proceedings of the 2018 ACM International Conference on Multimedia Retrieval (pp. 135–142). Association for Computing Machinery. https://doi.org/10.1145/3206025.3206037

Abstract

Background Heggli et al. (2021) noted that circadian rhythms and individual traits influence diurnal patterns in music preference. Depressive individuals often experience more negative moods in the evening, known as the evening-worsening effect (Rusting & Larsen, 1998). As music is often used for emotion focused-coping during such mood states (Stewart et al., 2019), depressive individuals’ music consumption in the evening are unclear. Aims This study explores the evening-worsening effect in music listening, examining if depressive individuals prefer low-valence, low-arousal music in the evening. Methods A survey of 290 Indians (M=20.35yrs, SD=2.08, 211 males) collected Kessler Psychological Distress scores (K10) and Healthy-Unhealthy music scale (HUMS) scores along with Spotify listening histories over one year. Additional measures like trait empathy and life satisfaction were collected but not analysed. Participants were classified as No-Risk (K10<20, n=114, 82 males) or At-Risk (K10≥29, n=81, 52 males) of depression (Andrews & Slade, 2001). Spotify valence and energy features for top 100-250 songs over the past 4-8 weeks from the time they filled in the survey were analyzed alongside timestamps. K-Means clustering divided the day into early hours, morning, afternoon and evening, and group differences in hourly valence and energy averages were assessed using two-way ANOVA and t-tests. Results At-Risk had significantly higher Unhealthy scores than No-Risk (t=8.15, p<0.001). Significant differences in valence and energy were found between At-Risk and No-Risk only in the afternoon (valence: t=-9.47, energy: t=-6.53) and evening (valence: t=-2.98, energy: t=-11.56)(all p≤ 0.05). Two-way ANOVA showed significant main effects of time, group, and their interaction for valence (time: F(3, 40)=22.86; group: F(1, 40)=12.36; interaction: F(3, 40)=6.65) and energy (time: F(3, 40)=23.22; group: F(1, 40)=98.73; interaction: F(3, 40)=13.57)(all p<0.001), consistent across the top 100-250 songs and the past 4-8 weeks of listening histories. Conclusions This study confirms diurnal patterns in music preferences, with At-Risk individuals favouring low-valence and low-energy music, especially in the afternoon and evening. These results align with Rusting and Larsen’s (1998) evening-worsening effect, linking depressive tendencies to heightened negative moods in the evening. The findings suggest potential maladaptive listening behaviors, needing further research.

Abstract

Background: Sensory Processing Sensitivity (SPS), a temperament trait reflecting deeper cognitive processing and heightened emotional/physiological reactivity, is often linked to emotional dysregulation and vulnerability to stress-related problems. SPS includes three subscales: Ease of Excitation (EOE) reflects emotional sensitivity to internal (e.g., thoughts, bodily states) and external (e.g., social demands) stimuli; Low Sensory Threshold (LST) reflects sensitivity to intense sensory input; and Aesthetic Sensitivity (AES) denotes appreciation for subtleties (Smolewska et al., 2006). Music can sometimes contribute to maladaptive outcomes like rumination, avoidance coping, and amplification of negative affect (Alluri et al., 2022). Highly sensitive persons, especially with elevated EOE, may be more prone to such patterns due to their emotional reactivity. However, SPS’s interactive role in maladaptive music use remains underexplored. Aim: This study investigates the relationship between SPS and maladaptive music use, hypothesizing that individuals with high EOE, particularly those sensitive to external demands (External EOE), are more prone to unhealthy music engagement. Network analysis was employed to reveal how sensitivity traits, distress, and empathy interact in shaping music use patterns. Methods: A total of 680 Indian participants(mean age=20.69, s.d.=1.9; 180 females) completed a battery of self-report measures including Highly Sensitive Persons Scale (HSPS) for SPS traits, the Kessler Psychological Distress Scale (K10) (Andrews & Slade, 2001), Interpersonal Reactivity Index (IRI) for empathy traits (Keaton, 2017), and Healthy-Unhealthy Music Scale (Saarikallio et al., 2015) assessing adaptive and maladaptive music use. The IRI included four subscales: Empathic Concern (EC), Perspective Taking (PT), Personal Distress (PD), and Fantasy Scale (FS), capturing both cognitive and affective empathy. Using exploratory factor analysis we split EOE into Internal and External factors. Regularized Partial Correlation Networks (RPCNs) and Bayesian Networks were constructed to model interactions and infer relationships among variables. Results: Unhealthy music use correlated with External EOE(r=0.25) while Healthy music use correlated with AES(r=0.32) among SPS traits. RPCNs identified External EOE as a central node in the network(strength=0.69; betweenness=116), bridging SPS traits with distress-related personal feelings(PD from IRI), psychological symptoms (K10), and Unhealthy music use. Bayesian Networks confirmed K10 as a direct predictor of Unhealthy use(B=0.50), with PD predicting K10(B=0.31) and External EOE predicting PD(B=0.50). Mediation analysis revealed that K10 significantly mediated the pathway from External EOE to Unhealthy music use, accounting for 55% of the total effect (all p<0.01), suggesting distress amplifies the impact of emotional sensitivity on maladaptive behaviors. Discussion: While music can support emotion regulation, our results suggest that highly sensitive persons experiencing distress may turn to music as a maladaptive coping strategy to escape overwhelming external demands. In India, where population density and sensory overstimulation are high, such patterns may be intensified. These findings add to the emerging literature on trait-environment interactions in music behavior and suggest that understanding EOE may inform future interventions or wellness strategies tailored to individual sensitivity profiles.

Abstract

Sensory processing sensitivity (SPS), a trait marked by heightened reactivity to stimuli, is linked to emotional dysregulation and stress-related problems. Its subscales include Ease of Excitation that reflects emotional sensitivity to internal and external demands, Low Sensory Threshold reflects sensory overload susceptibility, and Aesthetic Sensitivity denotes appreciation for subtleties. While music can sometimes amplify maladaptive outcomes (rumination, avoidance), SPS’s role in such behaviours, especially in non-Western contexts, remains underexplored. This study examines how SPS drives maladaptive music listening in 673 Indian adults. Network analysis and structural equation modelling revealed sensitivity to external demands elevated psychological distress which in turn predicted maladaptive music use. Reactivity to internal demands may reduce maladaptive music use but exacerbated it when mediated by external demands, reflecting preference for avoidant coping in Indian context. Findings emphasize SPS’s role in maladaptive behaviours, demonstrating how sensory reactivity interacts with traits to shape distress-driven music use as an emotional regulation mechanism.

Abstract

Personalised Interactive Music Systems (PIMS) are emerging as promising devices for enhancing physical activity and exercise outcomes. By leveraging real-time data and adaptive technologies, PIMS align musical features, such as tempo and genre with users’ physical activity patterns, including frequency and intensity, enhancing their overall experience. This systematic review and exploratory meta-analysis evaluates the effectiveness of PIMS across physical, psychophysical, and affective domains. Searches across nine databases identified 18 eligible studies, of which six (comprising 17 intervention arms) contained sufficient data for meta-analysis. Random-effects meta-analyses and meta-regression were performed to assess outcomes for physical activity levels, physical exertion, ratings of perceived exertion (RPE), and affective valence. Results showed significant improvements in physical activity levels (g = 0.49, CI [0.07, 0.91], p = .02, k = 4) and affective valence (g = 1.68, CI [0.15, 3.20], p = .03, k = 4), with faster music tempo identified as a significant moderator (p = 0.04). No significant effects were observed for RPE (g = 0.72, CI [-0.14, 1.59], p = .10, k = 3) or physical exertion (g = 0.79, CI [-0.64, 2.10], p = .28, k = 5). Substantial heterogeneity and limited study quality indicate the need for more robust, randomised controlled trials to establish the efficacy of PIMS in diverse populations.

Abstract

Deep learning models are capable of complex auditory processing tasks such as keyword spotting, genre classification, and audio captioning, yet remain opaque. While several works have explored interpretability of neural networks for computer vision and natural language processing, the audio modality has been largely ignored. In this paper, we study the behavior of the audio CNN encoder used in the contrastively trained language-audio model, CLAP. In the domain of music and human speech sounds, we localize and identify the layers of the network that perform well on tasks of varying complexity, sometimes even outperforming the model's final outputs. Digging deeper, we also localize specific dataset classes to neuron clusters within a layer and analyze a cluster’s contribution to the model’s discriminability for that class. To perform these analyses, we propose an automated framework that can leverage a small dataset of a few thousand samples to evaluate and score neuron clusters for their role in classification. Our findings provide insights into the hierarchical nature of representations in audio CNNs, paving the way for improved interpretability of audio model.

Abstract

A method for identifying a soundtrack for a digital book by automatically aligning the digital book and the soundtrack with a movie using a movie adaptation technique is provided. The method includes (i) receiving media content through a user device that includes the digital book, the soundtrack, and the movie,(ii) segmenting (a) the chapters of the digital book into paragraph segments,(b) the movie into scene boundaries, and (c) tracks of the soundtrack into cohesive track segments,(iii) aligning the scene boundaries with the paragraph segments to generate aligned paragraph segments,(iv) aligning a background soundtracks of the movie with the cohesive track segments of the soundtrack using a majority key and a minority key of the cohesive track segments and the background soundtracks to generate aligned cohesive track segments and (v) aligning the aligned paragraph segments with the aligned cohesive