Abstract

How does the brain represent different modes of information? Can we design a system that automatically understands what the user is thinking? Such questions can be answered by studying brain recordings like functional magnetic resonance imaging (fMRI). As a first step, the neuroscience community has contributed several large cognitive neuroscience datasets related to passive reading/listening/viewing of concept words, narratives, pictures and movies. Encoding and decoding models using these datasets have also been proposed in the past two decades. These models serve as additional tools for basic research in cognitive science and neuroscience. Encoding models aim at generating fMRI brain representations given a stimulus automatically. They have several practical applications in evaluating and diagnosing neurological conditions and thus also help design therapies for brain damage. Decoding models solve the inverse problem of reconstructing the stimuli given the fMRI. They are useful for designing brain-machine or brain-computer interfaces. Inspired by the effectiveness of deep learning models for natural language processing, computer vision, and speech, recently several neural encoding and decoding models have been proposed. In this survey, we will first discuss popular representations of language, vision and speech stimuli, and present a summary of neuroscience datasets. Further, we will review popular deep learning based encoding and decoding architectures and note their benefits and limitations. Finally, we will conclude with a brief summary and discussion about future trends

Abstract

Syntactic parsing is the task of assigning a syntactic structure to a sentence. There are two popular syntactic parsing methods: constituency and dependency parsing. Recent works have used syntactic embeddings based on constituency trees, incremental top-down parsing, and other word syntactic features for brain activity prediction given the text stimuli to study how the syntax structure is represented in the brain’s language network. However, the effectiveness of dependency parse trees or the relative predictive power of the various syntax parsers across brain areas, especially for the listening task, is yet unexplored. In this study, we investigate the predictive power of the brain encoding models in three settings: (i) individual performance of the constituency and dependency syntactic parsing based embedding methods, (ii) efficacy of these syntactic parsing based embedding methods when controlling for basic syntactic signals, (iii) relative effectiveness of each of the syntactic embedding methods when controlling for the other. Further, we explore the relative importance of syntactic information (from these syntactic embedding methods) versus semantic information using BERT embeddings. We find that constituency parsers help explain activations in the temporal lobe and middle-frontal gyrus, while dependency parsers better encode syntactic structure in the angular gyrus and posterior cingulate cortex. Although semantic signals from BERT are more effective compared to any of the syntactic features or embedding methods, syntactic embedding methods explain additional v

Abstract

Deep Reinforcement learning is beginning to be useful for studying neural representations in the brain because of its ability to combine decision-making and representation. Here, we use it to study a dot motion perceptual decision-making task in a high-dimensional setting where the inputs are akin to those used in psychological experiments. This end-to-end model gives a unique insight into how these networks solve the task providing a background on how the brain could solve this task. We find that the network can show properties similar to the middle temporal visual area (MT) in the brain, which code for direction and motion strength. We find the emergence of direction selectivity purely through reward-based training and graded firing coding motion strength and make a testable prediction that the MT population would also have coherence-selective neurons.

Abstract

One of the challenges in stroke rehabilitation is to identify bio-markers that correlate with amelioratory changes in the recovery of brain function that can be verified by a clinician. For strokes related to upper extremity, clinicians use Fugl Meyer Assessment - Upper Extremity (FMA-UE) score for verification. We hypothesize that even before clinical measures of function recovery (FMA-UE) show facilitatory changes, structural and functional changes in the brain connectome indicate early changes in stroke patients. Toward establishing such early bio-markers of rehabilitation related changes in the brain plasticity, we propose to use graph theoretic measures on the structural (SC) and functional connectivity (FC) matrices. We used longitudinal multi-modal neuroimaging data acquired within 1 to 6 months of the onset of stroke and after 3 months of rehabilitation for 15 acute ischemic stroke subjects with deficit in motor

Abstract

Autism spectrum disorder (ASD) is a neurodevelopmental disorder that predominantly occurs in children. Previous brain research in ASD has mainly studied biomarkers based on the functional connectivity characterized by the correlation of static temporal signals. However, brain connectivity is dynamic and varies extensively among brain states. The main aim of the paper is to understand the fundamental group differences between ASD patients and typically developing (TD) subjects using dynamic functional connectivity (dFNC) analysis. In this study, we investigated the dFNC between 53 independent components among 188 ASD and 195 TD subjects. We estimated dFNC using sliding window-based approaches and identified four distinct dynamic states through hard-clustering analysis. Hyper-connectivity within the cognitive control domain, between cognitive control and default mode network, has been …

Abstract

Reinforcement Learning (RL) models constitute an important subset of models used in studying many facets of human learning including Motor Sequence Learning. However, conventional action selection in RL models such as softmax-based choice rules lack biological plausibility and do not offer mechanistic explanations. Furthermore, they also do not use response time data in model fitting, which can be indicative of the difference in value of alternate choices as perceived by subjects. Evidence Accumulation Models(EAM) such as Linear Ballistic Accumulators(LBA) provide a solution to many of the above problems. In this study, we use RL algorithms integrated with an LBA model to model human behaviour in the Grid-Sailing Task. The task involves navigating a grid to reach a goal position where the participant can choose from three possible actions. We fit RLLBA models using three different RL algorithms: a …

Abstract

Cooperative human-human communication becomes challenging when restrictions such as difference in communication modality and limited time are imposed. We use the popular cooperative social game Pictionary as an online multimodal test bed to explore the dynamics of human-human interactions in such settings. As a part of our study, we identify attributes of player interactions that characterize cooperative gameplay. We found stable and role-specific playing style components that are independent of game difficulty. In terms of gameplay and the larger context of cooperative partially observable communication, our results suggest that too much interaction or unbalanced interaction negatively impacts game success. Additionally

Abstract

Individuals choose varying music listening strategies to fulfill particular mood-regulation goals. However, ineffective musical choices and a lack of cognizance of the effects thereof can be detrimental to their well-being and may lead to adverse outcomes like anxiety or depression. In our study, we use the social media platform Reddit to perform a large-scale analysis to unearth the several music-mediated mood-regulation goals that individuals opt for in the context of depression. A mixed-methods approach involving natural language processing techniques followed by qualitative analysis was performed on all music-related posts to identify the various music-listening strategies and group them into healthy and unhealthy associations. Analysis of the music content (acoustic features and lyrical themes) accompanying healthy and unhealthy associations showed significant differences. Individuals resorting to unhealthy strategies gravitate towards low-valence tracks. Moreover, lyrical themes associated with unhealthy strategies incorporated tracks with low optimism, high blame, and high self-reference. Our findings demonstrate that being mindful of the objectives of using music, the subsequent effects thereof, and aligning both for well-being outcomes is imperative for comprehensive understanding of the effectiveness of music

Abstract

Code-mixing refers to the phenomenon of using two or more languages interchangeably within a speech or discourse context. This practice is particularly prevalent on social media platforms, and determining the embedded affects in a code-mixed sentence remains as a challenging problem. In this submission we describe our system for WASSA 2023 Shared Task on Emotion Detection in English-Urdu code-mixed text. In our system we implement a multiclass emotion detection model with label space of 11 emotions. Samples are code-mixed English-Urdu text, where Urdu is written in romanised form. Our submission is limited to one of the subtasks - Multi Class classification and we leverage transformer-based Multilingual Large Language Models (MLLMs), XLM-RoBERTa and Indic-BERT. We fine-tune MLLMs on the released data splits, with and without pre-processing steps (translation to english), for classifying texts into the appropriate emotion category. Our methods did not surpass the baseline, and our submission is ranked sixth overall.

Abstract

Task-oriented dialogue research has mainly focused on a few popular languages like English and Chinese, due to the high dataset creation cost for a new language. To reduce the cost, we apply manual editing to automatically translated data. We create a new multilingual benchmark, X-RiSAWOZ, by translating the Chinese RiSAWOZ to 4 languages: English, French, Hindi, Korean; and a code-mixed EnglishHindi language. X-RiSAWOZ has more than 18,000 human-verified dialogue utterances for each language, and unlike most multilingual prior work, is an end-to-end dataset for building fully-functioning agents. The many difficulties we encountered in creating X-RiSAWOZ led us to develop a toolset to accelerate the post-editing of a new language dataset after translation. This toolset improves machine translation with a hybrid entity alignment technique that combines neural with