Abstract

Architecture Knowledge Management (AKM) is crucial for maintaining current and comprehensive software Architecture Knowledge (AK) in a software project. However AKM is often a laborious process and is not adopted by developers and architects. While LLMs present an opportunity for automation, a naive, single-prompt approach is often ineffective, constrained by context limits and an inability to grasp the distributed nature of architectural knowledge. To address these limitations, we propose an Agentic approach for AKM, AgenticAKM, where the complex problem of architecture recovery and documentation is decomposed into manageable sub-tasks. Specialized agents for architecture Extraction, Retrieval, Generation, and Validation collaborate in a structured workflow to generate AK. To validate we made an initial instantiation of our approach to generate Architecture Decision Records (ADRs) from code repositories. We validated our approach through a user study with 29 repositories. The results demonstrate that our agentic approach generates better ADRs, and is a promising and practical approach for automating AKM.

Abstract

Context. The rise of Large Language Models (LLMs) has led to their widespread adoption in development pipelines.
Goal. We empirically assess the energy efficiency of Python code generated by LLMs against human-written code and code developed by a Green software expert.
Method. We test 363 solutions to 9 coding problems from the EvoEval benchmark using 6 widespread LLMs with 4 prompting techniques, and comparing them to human-developed solutions. Energy consumption is measured on three different hardware platforms: a server, a PC, and a Raspberry Pi for a total of ≈881h (36.7 days).
Results. Human solutions are 16% more energy-efficient on the server and 3% on the Raspberry Pi, while LLMs outperform human developers by 25% on the PC. Prompting does not consistently lead to energy savings, where the most energy-efficient prompts vary by hardware platform. The code developed by a Green software expert is consistently more energy-efficient by at least 17% to 30% against all LLMs on all hardware platforms.
Conclusions. Even though LLMs exhibit relatively good code generation capabilities, no LLM-generated code was more energy-efficient than that of an experienced Green software developer, suggesting that as of today there is still a great need of human expertise for developing energy-efficient Python code.

Abstract

Large Language Models (LLMs) are increasingly used in software engineering, yet their role in architectural refactoring, particularly in decomposing monolithic systems into microservices, remains largely unexplored, even though such decomposition, when necessary, is difficult and error-prone, especially when identifying meaningful service boundaries.
This exploratory study investigates how LLMs can support architects through a multi-step decomposition process that applies seven summarisation strategies and four LLMs to generate architectural views and candidate decompositions. We assess these decompositions using six established metrics across four open-source systems that provide both monolithic and microservice versions, enabling comparison with reference architectures.
Results show that LLM-based decompositions outperform traditional techniques on Cyclic Independence (+22.7), approaching reference architectures and indicating better technical dependency management. Traditional automated approaches remain stronger at recovering the ground-truth structure (MoJoFM +24.1), team alignment (TC +11.0), and domain-oriented boundaries (DI +10.7). No significant differences appear for Code Modularity or Business Context Purity, suggesting that LLMs produce clean decompositions with few cyclic dependencies but still struggle to reproduce the exact target decomposition.

Abstract

Context: Architecture views are essential for soft-
ware architecture documentation, yet their manual creation is labor-intensive and often leads to outdated artifacts. As systems grow in complexity, the automated generation of views from source code becomes increasingly valuable.
Goal: We empirically evaluate the ability of LLMs and agentic approaches to generate architecture views from source code.
Method: We analyze 340 open-source repositories across 13 experimental configurations using 3 LLMs with 3 prompting techniques and 2 agentic approaches, yielding 4,137 generated views. We evaluate the generated views by comparing them with the ground-truth using a combination of automated metrics complemented by human evaluations.
Results: Prompting strategies offer marginal improvements. Few-shot prompting reduces clarity failures by 9.2% compared to zero-shot baselines. The custom agentic approach consistently outperforms the general purpose agent, achieving the best clarity
(22.6% failure rate) and level of detail success (50%).
Conclusions: LLM and agentic approaches demonstrate capabilities in generating syntactically valid architecture views. However, they consistently exhibit granularity mismatches, operating at the code level rather than architectural abstractions. This suggests
that there is still a need for human expertise, positioning LLMs and agents as assistive tools rather than autonomous architects.

Abstract

This study investigates the performance, energy efficiency, and resource consumption of four agentic issue resolution frameworks (SWE-Agent, OpenHands, Mini SWE Agent, and AutoCodeRover) when constrained to Small Language Models (SLMs). Results indicate that framework architecture is the primary driver of energy consumption, but current frameworks fail to operate efficiently with SLMs, often leading to unproductive reasoning loops and low task resolution rates.

Abstract

Energy efficiency has emerged as a vital attribute of software quality,
with significant implications for both environmental sustainability
and operational costs. However, existing profiling tools operate
only at runtime and coarse granularity, typically capturing energy
at the process or method level. Such tools fail to expose how small
code blocks, such as functions, loops, and conditionals, contribute to
energy consumption, preventing developers from reasoning about
and comparing the energy efficiency of programming constructs
during design-time.
To address this gap, we propose EnCoDe, a methodology for
fine-grained, design-time energy estimation, with the following
key contributions: (1) PowerLens, a novel measurement method-
ology that achieves reliable sub-millisecond energy readings for
small code blocks; (2) Extensive empirical study on code blocks
extracted from over 18,000 Python programs, uncovering linear and
non-linear relationships between energy consumption and static
code features such as structural, complexity, density, and contextual
characteristics, resulting in a first-of-its-kind fine-grained dataset;
and (3) Predictive modeling, in which machine learning models
are trained on these features to accurately estimate and classify
block-level energy consumption at design-time. Our results demon-
strate stable, reproducible block-level estimations, with regressors
achieving 𝑅2 = 0.75 and classifiers achieving 80.6% accuracy in
identifying energy hotspots, enabling developers to localize and
address inefficient code regions early in the development process
without execution

Abstract

LLMs have advanced code generation, but their use for generating microservices with explicit dependencies and API contracts remains understudied. We examine whether AI agents can generate functional microservices and how different forms of contextual information influence their performance. We assess 144 generated microservices across 3 agents, 4 projects, 2 prompting strategies, and 2 scenarios. Incremental generation operates within existing systems and is evaluated with unit tests. Clean state generation starts from requirements alone and is evaluated with integration tests. We analyze functional correctness, code quality, and efficiency. Minimal prompts outperformed detailed ones in incremental generation, with 50-76% unit test pass rates. Clean state generation produced higher integration test pass rates (81-98%), indicating strong API contract adherence. Generated code showed lower complexity than human baselines. Generation times varied widely across agents, averaging 6-16 minutes per service. AI agents can produce microservices with maintainable code, yet inconsistent correctness and reliance on human oversight show that fully autonomous microservice generation is not yet achievable.

Abstract

Benchmarks for large language models (LLMs) have progressed from snippet-level function generation to repository-level issue resolution, yet they overwhelmingly target implementation correctness. Software architecture tasks remain under-specified and difficult to compare across models, despite their central role in maintaining and evolving complex systems. We present ArchBench, the first unified platform for benchmarking LLM capabilities on software architecture tasks. ArchBench provides a command-line tool with a standardized pipeline for dataset download, inference with trajectory logging, and automated evaluation, alongside a public web interface with an interactive leaderboard. The platform is built around a plugin architecture where each task is a self-contained module, making it straightforward for the community to contribute new architectural tasks and evaluation results. We use the term LLMs broadly to encompass generative AI (GenAI) solutions for software engineering, including both standalone models and LLM-based coding agents equipped with tools. Both the CLI tool and the web platform are openly available to support reproducible research and community-driven growth of architectural benchmarking.

Abstract

Recent advances in agentic AI have shifted the focus from standalone Large Language Models (LLMs) to integrated systems that combine LLMs with tools, memory, and other agents to perform complex tasks. These multi-agent architectures enable coordinated reasoning, planning, and execution across diverse domains, allowing agents to collaboratively automate complex workflows. Despite these advances, evaluation and assessment of LLM agents and the multi-agent systems they constitute remain a fundamental challenge. Although various approaches have been proposed in the software engineering literature for evaluating conventional software components, existing methods for AI-based systems often overlook the non-deterministic nature of models. This non-determinism introduces behavioral uncertainty during execution, yet existing evaluations rely on binary task-completion metrics that fail to capture it. Evaluating agentic systems therefore requires examining additional dimensions, including the agent's ability to invoke tools, ingest and retrieve memory, collaborate with other agents, and interact effectively with its environment.

These challenges emerged during our ongoing industry collaboration with MontyCloud Inc., an industrial partner in the domain of Autonomous CloudOps, when we deployed an agentic system in production. These limitations surfaced during deployment, highlighting practical gaps in the current evaluation methods and the need for a systematic assessment of agent behavior beyond task outcomes. Informed by these observations and established definitions of agentic systems, we propose an end-to-end Agent Assessment Framework with four evaluation pillars encompassing LLMs, Memory, Tools, and Environment. We validate the framework on a representative Autonomous CloudOps use case, where experiments reveal behavioral deviations overlooked by conventional metrics, demonstrating its effectiveness in capturing runtime uncertainties.

Abstract

Existing sustainability assessment tools for carbon emissions produced by cloud infrastructure predominantly focus on postdeployment monitoring or require access to application code. To address this gap, we present a framework that enables early-stage estimation of carbon footprint from terraform configurations.