Abstract

Effective decision-making in complex environments with multi-discrete action spaces poses significant challenges for agent architectures, particularly in image-based settings. While Decision Transformers have shown promise in various domains, their performance often suffers in environments where agents must handle multi-discrete actions. Existing enhancements to Decision Transformer architectures have yet to address this critical issue, limiting their ability to support agents in learning robust policies in these environments. To address this gap, we propose Multi-State Action Tokenisation (M-SAT), a novel approach designed to improve agent decision-making by tokenising actions at the individual action level and incorporating auxiliary state information. This disentanglement of actions improves both the performance of agents and the interpretability of individual actions within attention layers, fostering better visibility into agent decision processes. Importantly, M-SAT facilitates the development of more interpretable and transparent agents capable of making complex decisions in dynamic environments involving multi-discrete action spaces. We evaluate M-SAT on the challenging ViZDoom environments, focusing on scenarios with multi-discrete action spaces and image-based observations, such as Deadly Corridor, My Way Home and Death Match. Our approach demonstrates superior performance compared to baseline Decision Transformers, with no additional data or significant computational overheads. Furthermore, we observe that M-SAT does not require positional encoding to achieve high performance, with its removal occasionally leading to further improvements. These findings suggest that M-SAT enables more efficient and interpretable agent-based decision-making in multi-discrete action spaces.

Abstract

Moving Target Defense (MTD) has emerged as a proactive and dynamic framework to counteract evolving cyber threats. Traditional MTD approaches often rely on assumptions about the attacker’s knowledge and behavior. However, real-world scenarios are inherently more complex, with adaptive attackers and limited prior knowledge of their payoffs and intentions. This paper introduces a novel approach to MTD using a Markov Decision Process (MDP) model that does not rely on predefined attacker payoffs. Our framework integrates the attacker’s real-time responses into the defender’s MDP using a dynamic Bayesian Network. By employing a factored MDP model, we provide a comprehensive and realistic representation of the system and its dependencies. We also incorporate incremental updates to an attack response predictor as new data emerges, ensuring an adaptive and robust defense mechanism. Additionally, we consider the costs of switching configurations in MTD, integrating them into the reward structure to balance execution and defense costs. We first highlight the challenges of the problem through a theoretical negative result on regret. However, our empirical evaluations, conducted in network and web application settings, demonstrate the framework’s effectiveness in environments characterized by high uncertainty and dynamically changing attack surfaces.

Abstract

Emergency vehicles (EVs) perform a critical task of attending medical emergencies and delay in their operations can result in loss of lives to long term or permanent health implications. Therefore, it is very important to design strategies that can reduce the delay of EVs caused by slow moving traffic. Most of the existing work on this topic focuses on assignment and dispatch of EVs from different base stations to hospitals or finding the appropriate routes from dispatch location to hospital. However, these works ignore the effect of lane changes when EV is travelling on a stretch of a road. In this work, we focus on lane level dynamics for EV traversal and showcase that a pro-active picking of lanes can result in significant reductions in traversal time. In particular, we design a Reinforcement Learning (RL) model to compute the most optimal lane for an EV to travel at each timestep. We propose RLLS (Reinforcement Learning based Lane Search) algorithm for a general purposes EV trave

Abstract

In literature, various time-frequency representation methods were investigated for automatic detection of voice disorders. Stockwell-Transform (S-Transform) provides good time-frequency localization; hence, it may efficiently capture the voice disorder related information from speech signal. With this motivation, we investigated different variants of S-Transform for the classification of voice disorders. This study proposed the S-Transform based cepstral coefficients for voice disorder detection and assessment. The performance of the proposed feature was compared with baseline features on SVD and HUPA databases. Compared to baseline features, proposed features performed best in terms of classification accuracy of 80.2% and 79.8% on HUPA and SVD databases, respectively for voice disorder detection task. Also, the proposed features performed better in case of assessment task. Further, the experimental results reveal that combining cepstral coefficients derived from S-Transform with baseline features improved the performance of proposed systems by 8% and 4% for detection and assessment tasks, respectively which highlights complementary nature of the explored features. We also analysed the effectiveness of S-Transform based spectral representation in capturing the acoustic characteristics for various voice qualities like breathiness, harshness, creakiness, and falsetto phonations. This representation was also compared with other time-frequency based methods such as STFT, ZTW and SFF. It was observed that S-Transform effectively captures the acoustic variations associated with different voice qualities compared to other baseline methods, which may be due to better spectro-temporal resolution offered by the S-Transform.

Abstract

In safety-critical RL settings, the inclusion of an additional cost function is often favoured over the arduous task of modifying the reward function to ensure the agent's safe behaviour. However, designing or evaluating such a cost function can be prohibitively expensive. For instance, in the domain of self-driving, designing a cost function that encompasses all unsafe behaviours (eg, aggressive lane changes, risky overtakes) is inherently complex, it must also consider all the actors present in the scene making it expensive to evaluate. In such scenarios, the cost function can be learned from feedback collected offline in between training rounds. This feedback can be system generated or elicited from a human observing the training process. Previous approaches have not been able to scale to complex environments and are constrained to receiving feedback at the state level which can be expensive to collect. To this end, we introduce an approach that scales to more complex domains and extends beyond state-level feedback, thus, reducing the burden on the evaluator. Inferring the cost function in such settings poses challenges, particularly in assigning credit to individual states based on trajectory-level feedback. To address this, we propose a surrogate objective that transforms the problem into a state-level supervised classification task with noisy labels, which can be solved efficiently. Additionally, it is often infeasible to collect feedback for every trajectory generated by the agent, hence, two fundamental questions arise:(1) Which trajectories should be presented to the human? and (2) How many trajectories are necessary for effective learning?

Abstract

Preserving the privacy of preferences (or rewards) of a sequential decision-making agent when decisions are observable is crucial in many physical and cybersecurity domains. For instance, in wildlife monitoring, forest rangers must conduct surveillance without revealing animal locations to poachers. This paper addresses privacy preservation in planning over a sequence of actions in MDPs, where the reward function represents the preference structure to be protected. Observers can use Inverse RL (IRL) to learn these preferences, making this a challenging task. Current research on Differential Privacy (DP) in this setting fails to ensure a lower bound on the minimum expected reward and offers theoretical guarantees that are inadequate against IRL-based observers. To bridge this gap, we propose a novel approach rooted in the theory of deception. Deception includes two models: dissimulation (hiding the truth) and simulation (showing the wrong). As our first contribution, we theoretically demonstrate a significant privacy leak in the current dissimulation-based method. Our second contribution is a novel RL-based planning algorithm that uses simulation to effectively address these privacy concerns while ensuring a guarantee on the expected reward. Through experimentation on multiple benchmark problems, we show that our proposed approach outperforms existing methods in preserving the privacy of reward functions.

Abstract

Decision Transformers have demonstrated impressive performance in offline reinforcement learning, but their complex internal mechanisms remain challenging to interpret. This paper presents a comprehensive analysis of Decision Transformers(DT) trained on various MuJoCo environments using different interpretability techniques. Our analysis includes: (a) positional encoding (PE) analysis to assess the impact of temporal information on performance in different types of environments, (b) return-to-go (RTG) analysis to explore the variability in achieving specific returns, (c) embedding analysis revealing the models' ability to learn detailed representations of the environments, (d) attention visualizations to understand the role of attention in behaviour guidance, and (e) perturbation studies to test robustness to input noise. Our analysis reveals the extent to which positional information is necessary for achieving high performance in different MuJoCo environments and provides insights into the model's adaptability to varying task requirements. In addition, the embedding analysis shows hierarchical abstractions captured by the model across layers. Through these analyses, we provide novel insights into the decision-making process of transformers in continuous control domains. These insights are pivotal for designing more interpretable and reliable decision-making systems and deepen our comprehension of the capabilities and limitations of DT in complex tasks.

Abstract

Moving Target Defense (MTD) has emerged as a proactive and dynamic framework to counteract evolving cyber threats. Traditional MTD approaches often rely on assumptions about the attackers knowledge and behavior. However, real-world scenarios are inherently more complex, with adaptive attackers and limited prior knowledge of their payoffs and intentions. This paper introduces a novel approach to MTD using a Markov Decision Process (MDP) model that does not rely on predefined attacker payoffs. Our framework integrates the attackers real-time responses into the defenders MDP using a dynamic Bayesian Network. By employing a factored MDP model, we provide a comprehensive and realistic system representation. We also incorporate incremental updates to an attack response predictor as new data emerges. This ensures an adaptive and robust defense mechanism. Additionally, we consider the costs of switching configurations in MTD, integrating them into the reward structure to balance execution and defense costs. We first highlight the challenges of the problem through a theoretical negative result on regret. However, empirical evaluations demonstrate the frameworks effectiveness in scenarios marked by high uncertainty and dynamically changing attack landscapes.

Abstract

No Results Found

Abstract

Air pollution is a growing concern across the world. Road traffic is one of the major contributors to air pollution in urban areas. One of the approaches to solve this problem is to design a transportation policy that i) avoids extreme pollution in any area, ii) enables short transit times, and iii) makes effective use of the road capacities. Previous work to address this problem named MaxFlow-MCMC algorithm, proposed a novel sampling-based approach for this problem. In this work, we propose a significantly faster extension to the algorithm without compromising on the performance involving the following contributions: (a) We provide the first construction of a Markov Chain to sample a set of k-optimal max flow solutions directly from a planar graph. (b) We simulate traffic on large-scale real-world roadmaps using the SUMO traffic simulator. We observe a significant speed improvement in the range of 22 to 242 times in our experiments while obtaining lesser average pollution.