Add literature review protocol, searches and first snowball round

Six seeds snowballed (Semantic Scholar) plus logged keyword searches
(OpenAlex, arXiv): 2,127 unique records, 480 title/abstract screened.
AI first-pass suggestions are in their own column; decisions are pending.

Claude-Session: https://claude.ai/code/session_01S8KN7NuUCm8nXLmy6eDuVv

02-literature/search/results/L2-openalex-llm-slr-software-eng.csv

@@ -0,0 +1,101 @@
+source_id,arxiv,doi,year,title,first_author,n_authors,venue,citations,abstract
+W4402665833,,10.1145/3695988,2024,Large Language Models for Software Engineering: A Systematic Literature Review,Xinyi Hou,10,ACM Transactions on Software Engineering and Methodology,1026,"Large Language Models (LLMs) have significantly impacted numerous domains, including Software Engineering (SE). Many recent publications have explored LLMs applied to various SE tasks. Nevertheless, a comprehensive understanding of the application, effects, and possible limitations of LLMs on SE is still in its early stages. To bridge this gap, we conducted a Systematic Literature Review (SLR) on LLM4SE, with a particular focus on understanding how LLMs can be exploited to optimize processes and outcomes. We selected and analyzed 395 research articles from January 2017 to January 2024 to answer four key Research Questions (RQs). In RQ1, we categorize different LLMs that have been employed in SE tasks, characterizing their distinctive features and uses. In RQ2, we analyze the methods used in data collection, pre-processing, and application, highlighting the role of well-curated datasets for successful LLM for SE implementation. RQ3 investigates the strategies employed to optimize and evaluate the performance of LLMs in SE. Finally, RQ4 examines the specific SE tasks where LLMs have shown success to date, illustrating their practical contributions to the field. From the answers to these RQs, we discuss the current state-of-the-art and trends, identifying gaps in existing research, and highlighting promising areas for future study. Our artifacts are publicly available at https://github.com/security-pride/LLM4SE_SLR ."
+W4386081573,10.48550/arxiv.2308.10620,10.48550/arxiv.2308.10620,2023,Large Language Models for Software Engineering: A Systematic Literature Review,Xinyi Hou,10,arXiv (Cornell University),114,"Large Language Models (LLMs) have significantly impacted numerous domains, including Software Engineering (SE). Many recent publications have explored LLMs applied to various SE tasks. Nevertheless, a comprehensive understanding of the application, effects, and possible limitations of LLMs on SE is still in its early stages. To bridge this gap, we conducted a systematic literature review (SLR) on LLM4SE, with a particular focus on understanding how LLMs can be exploited to optimize processes and outcomes. We select and analyze 395 research papers from January 2017 to January 2024 to answer four key research questions (RQs). In RQ1, we categorize different LLMs that have been employed in SE tasks, characterizing their distinctive features and uses. In RQ2, we analyze the methods used in data collection, preprocessing, and application, highlighting the role of well-curated datasets for successful LLM for SE implementation. RQ3 investigates the strategies employed to optimize and evaluate the performance of LLMs in SE. Finally, RQ4 examines the specific SE tasks where LLMs have shown success to date, illustrating their practical contributions to the field. From the answers to these RQs, we discuss the current state-of-the-art and trends, identifying gaps in existing research, and flagging promising areas for future study. Our artifacts are publicly available at https://github.com/xinyi-hou/LLM4SE_SLR."
+W4414594130,,10.1145/3769676,2025,Large Language Models for Cyber Security: A Systematic Literature Review,Hanxiang Xu,9,ACM Transactions on Software Engineering and Methodology,185,"The rapid advancement of Large Language Models (LLMs) has opened up new opportunities for leveraging artificial intelligence in a variety of application domains, including cybersecurity. As the volume and sophistication of cyber threats continue to grow, there is an increasing need for intelligent systems that can automatically detect vulnerabilities, analyze malware, and respond to attacks. In this survey, we conduct a comprehensive review of the literature on the application of LLMs in cybersecurity (LLM4Security). By comprehensively collecting over 40K relevant papers and systematically analyzing 185 papers from top security and software engineering venues, we aim to provide a holistic view of how LLMs are being used to solve diverse problems across the cybersecurity domain. Through our analysis, we identify several key findings. First, we observe that LLMs are being applied to an expanding range of cybersecurity tasks, including vulnerability detection, malware analysis, and network intrusion detection. Second, we analyze application trends of different LLM architectures (such as encoder-only, encoder-decoder, and decoder-only) across security domains. Third, we identify increasingly sophisticated techniques for adapting LLMs to cybersecurity, such as advanced fine-tuning, prompt engineering, and external augmentation strategies. A significant emerging trend is the use of LLM-based autonomous agents, which represent a paradigm shift from single-task execution to orchestrating complex, multi-step security workflows. Furthermore, we find that the datasets used for training and evaluating LLMs are often limited, highlighting the need for more comprehensive datasets and the use of LLMs for data augmentation. Finally, we discuss the main challenges and opportunities for future research, including the need for more interpretable models, addressing the inherent security risks of LLMs, and their potential for proactive defense. Overall, our survey provides a comprehensive overview of the current state-of-the-art in LLM4Security and identifies several promising directions for future research. We believe that the insights and findings presented in this survey will contribute to the growing body of knowledge on the application of LLMs in cybersecurity and provide valuable guidance for researchers and practitioners working in this field."
+W7122824220,,10.5281/zenodo.18215692,2026,Large Language Models for Software Engineering: A Systematic Literature Review,Tahomina Sultana,1,Zenodo (CERN European Organization for Nuclear Research),0,
+W4413394768,,10.1109/access.2025.3601206,2025,A Systematic Literature Review of Hallucinations in Large Language Models,Christian Woesle,3,IEEE Access,22,"This review systematically maps research on hallucinations in large language models using a descriptive scheme that links model outputs to four system architectures: unaugmented generation, post-hoc reactive validation, proactive detection-and-mitigation, and fully integrated detection-and-mitigation designs. Our methodology for this systematic review follows the PRISMA guidelines to ensure transparency and reproducibility. We searched IEEE Xplore, ACM Digital Library, and ScienceDirect for studies published between 2015 and January 2025 and extracted 125 peer-reviewed papers across nine application domains. Quantitative analysis shows that question answering and multimodal tasks account for 48% of all papers, whereas software engineering, educational technology, and autonomous systems are underexplored. Although 87.5% of the studies rely on additional reactive or proactive defenses, only 8.8% implement integrated architecture-level safeguards, revealing a critical gap in unified and dynamic architectures. The resulting classification matrix and domain map provide a diagnostic tool for locating blind spots and comparing architectural maturity. Three actionable priorities emerge: develop integrated reasoning-and-verification loops that pre-empt hallucinations; transfer proven causal-intervention and multi-agent validation pipelines to high-stakes, under-represented domains and benchmark them under real conditions; and build modular, cross-domain evaluation frameworks that isolate the contribution of individual mitigation components and support ablation studies. By consolidating fragmented evidence and quantifying architecture-domain imbalances, this review establishes a traceable foundation for engineering reliable, explainable, and domain-adaptable countermeasures to hallucinations in generative language technology."
+W4396821716,10.48550/arxiv.2405.04760,10.48550/arxiv.2405.04760,2024,Large Language Models for Cyber Security: A Systematic Literature Review,HanXiang Xu,9,arXiv (Cornell University),27,"The rapid advancement of Large Language Models (LLMs) has opened up new opportunities for leveraging artificial intelligence in a variety of application domains, including cybersecurity. As the volume and sophistication of cyber threats continue to grow, there is an increasing need for intelligent systems that can automatically detect vulnerabilities, analyze malware, and respond to attacks. In this survey, we conduct a comprehensive review of the literature on the application of LLMs in cybersecurity~(LLM4Security). By comprehensively collecting over 40K relevant papers and systematically analyzing 185 papers from top security and software engineering venues, we aim to provide a holistic view of how LLMs are being used to solve diverse problems across the cybersecurity domain. Through our analysis, we identify several key findings. First, we observe that LLMs are being applied to an expanding range of cybersecurity tasks, including vulnerability detection, malware analysis, and network intrusion detection. Second, we analyze application trends of different LLM architectures (such as encoder-only, encoder-decoder, and decoder-only) across security domains. Third, we identify increasingly sophisticated techniques for adapting LLMs to cybersecurity, such as advanced fine-tuning, prompt engineering, and external augmentation strategies. A significant emerging trend is the use of LLM-based autonomous agents, which represent a paradigm shift from single-task execution to orchestrating complex, multi-step security workflows."
+W4415088384,10.48550/arxiv.2509.11446,10.48550/arxiv.2509.11446,2025,Large Language Models (LLMs) for Requirements Engineering (RE): A Systematic Literature Review,Mohammad Amin Zadenoori,5,arXiv (Cornell University),9,"Large Language Models (LLMs) are finding applications in numerous domains, and Requirements Engineering (RE) is increasingly benefiting from their capabilities to assist with complex, language-intensive tasks. This paper presents a systematic literature review of 74 primary studies published between 2023 and 2024, examining how LLMs are being applied in RE. The study categorizes the literature according to several dimensions, including publication trends, RE activities, prompting strategies, and evaluation methods. Our findings indicate notable patterns, among which we observe substantial differences compared to previous works leveraging standard Natural Language Processing (NLP) techniques. Most of the studies focus on using LLMs for requirements elicitation and validation, rather than defect detection and classification, which were dominant in the past. Researchers have also broadened their focus and addressed novel tasks, e.g., test generation, exploring the integration of RE with other software engineering (SE) disciplines. Although requirements specifications remain the primary focus, other artifacts are increasingly considered, including issues from issue tracking systems, regulations, and technical manuals. The studies mostly rely on GPT-based models, and often use Zero-shot or Few-shot prompting. They are usually evaluated in controlled environments, with limited use in industry settings and limited integration in complex workflows. Our study outlines important future directions, such as leveraging the potential to expand the influence of RE in SE, exploring less-studied tasks, improving prompting methods, and testing in real-world environments. Our contribution also helps researchers and practitioners use LLMs more effectively in RE, by providing a list of identified tools leveraging LLMs for RE, as well as datasets."
+W4408867984,,10.21203/rs.3.rs-5589929/v1,2025,Large Language Model for Requirements Engineering: A Systematic Literature Review,Javed Ali Khan,3,Research Square,14,
+W4415333602,10.48550/arxiv.2506.15227,10.48550/arxiv.2506.15227,2025,Large Language Models for Unit Testing: A Systematic Literature Review,Quanjun Zhang,6,arXiv (Cornell University),2,"Unit testing is a fundamental practice in modern software engineering, with the aim of ensuring the correctness, maintainability, and reliability of individual software components. Very recently, with the advances in Large Language Models (LLMs), a rapidly growing body of research has leveraged LLMs to automate various unit testing tasks, demonstrating remarkable performance and significantly reducing manual effort. However, due to ongoing explorations in the LLM-based unit testing field, it is challenging for researchers to understand existing achievements, open challenges, and future opportunities. This paper presents the first systematic literature review on the application of LLMs in unit testing until March 2025. We analyze \numpaper{} relevant papers from the perspectives of both unit testing and LLMs. We first categorize existing unit testing tasks that benefit from LLMs, e.g., test generation and oracle generation. We then discuss several critical aspects of integrating LLMs into unit testing research, including model usage, adaptation strategies, and hybrid approaches. We further summarize key challenges that remain unresolved and outline promising directions to guide future research in this area. Overall, our paper provides a systematic overview of the research landscape to the unit testing community, helping researchers gain a comprehensive understanding of achievements and promote future research. Our artifacts are publicly available at the GitHub repository: https://github.com/iSEngLab/AwesomeLLM4UT."
+W7161417267,,10.1145/3815425,2026,A Systematic Literature Review on Detecting Software Vulnerabilities with Large Language Models,Sabrina Kaniewski,5,ACM Transactions on Software Engineering and Methodology,8,"The increasing adoption of Large Language Models (LLMs) in software engineering has sparked interest in their use for software vulnerability detection. However, the rapid development of this field has resulted in a fragmented research landscape, with diverse studies that are difficult to compare due to differences in, e.g., system designs and dataset usage. This fragmentation makes it difficult to obtain a clear overview of the state-of-the-art or compare and categorize studies meaningfully. In this work, we present a comprehensive systematic literature review (SLR) of LLM-based software vulnerability detection. We analyze 263 studies published between January 2020 and November 2025, categorizing them by task formulation, input representation, system architecture, and techniques. Further, we analyze the datasets used, including their characteristics, vulnerability coverage, and diversity. We present a fine-grained taxonomy of vulnerability detection approaches, identify key limitations, and outline actionable future research opportunities. By providing a structured overview of the field, this review improves transparency and serves as a practical guide for researchers and practitioners aiming to conduct more comparable and reproducible research. We publicly release all artifacts and maintain a living repository of LLM-based software vulnerability detection studies at https://github.com/hs-esslingen-it-security/Awesome-LLM4SVD ."
+W7167724220,,10.59431/jda.v5i1.810,2026,Large Language Model Integration in Desktop Applications: A Systematic Literature Review,Deyidi Mokoginta,1,Journal Dekstop Application (JDA),0,"The integration of Large Language Models (LLMs) into desktop applications has accelerated sharply over the past several years, yet no prior systematic review has examined this deployment setting as a distinct research domain — separate from web-based or mobile environments, each carrying different architectural constraints, privacy expectations, and user interaction patterns. This study maps, analyzes, and synthesizes the available scientific evidence on LLM architectures adopted in desktop implementations, the technical barriers arising during integration, the measurable effects on User Experience (UX), and the relative effectiveness of competing deployment strategies. A Systematic Literature Review (SLR) was conducted following PRISMA 2020 reporting guidelines, with research questions structured using the PICOC framework. Systematic searches across seven digital databases — IEEE Xplore, ACM Digital Library, Scopus, Web of Science, Google Scholar, arXiv, and Semantic Scholar — covering 2019–2024 yielded 1,200 initial articles, narrowed to 42 final articles through layered selection and quality appraisal using a modified CASP rubric. GPT-4 and the LLaMA family emerged as the dominant architectures, with model selection driven primarily by parameter scale and privacy requirements. Computational constraints, response latency, and data security ranked as the most serious technical barriers, with Retrieval-Augmented Generation (RAG) and model quantization as the most common mitigations. LLM integration was associated with measurable gains in task efficiency and user satisfaction in 73.8% of reviewed studies, though output predictability and user trust remained persistent challenges. A hybrid deployment approach — combining local models with cloud-based API access — emerged as the most common practice among production implementations. Unlike existing reviews addressing LLM capabilities in general software engineering contexts, this study provides the first systematic map of LLM integration specifically within the desktop application domain, offering concrete guidance for developers working at the intersection of artificial intelligence and desktop software engineering."
+W7199799894,,,2024,Utilizing Large Language Models to Update Systematic Literature Reviews: A Case Study on Time Pressure and Well-Being in Software Engineering,Jingyi Guo,1,Työväentutkimus Vuosikirja,0,"This thesis examines the application of Large Language Models (LLMs), including ChatGPT-4 and ChatGPT-4.o, to update systematic literature reviews (SLRs) with a focus on time pressure and well-being in software engineering. It aims to address the challenges of traditional SLRs, which often need more timeliness and automaticity in synthesizing the latest research findings. By conducting a case study, this research evaluates the efficiency, comprehensiveness, and accuracy of using LLMs to update SLRs, comparing the strengths and weaknesses of various LLMs in processing and analyzing vast amounts of literature. The study aims to provide insights into the potential of LLMs to revolutionize the methodology of SLRs, particularly in dynamically evolving fields such as software engineering."
+W4405640497,,10.1145/3708525,2024,"Efficient and Green Large Language Models for Software Engineering: Literature Review, Vision, and the Road Ahead",Jieke Shi,3,ACM Transactions on Software Engineering and Methodology,43,"Large Language Models (LLMs) have recently shown remarkable capabilities in various software engineering tasks, spurring the rapid growth of the Large Language Models for Software Engineering (LLM4SE) area. However, limited attention has been paid to developing efficient LLM4SE techniques that demand minimal computational cost, time, and memory resources, as well as green LLM4SE solutions that reduce energy consumption, water usage, and carbon emissions. This article aims to redirect the focus of the research community toward the efficiency and greenness of LLM4SE, while also sharing potential research directions to achieve this goal. It commences with a brief overview of the significance of LLM4SE and highlights the need for efficient and green LLM4SE solutions. Subsequently, the article presents a vision for a future where efficient and green LLM4SE revolutionizes the LLM-based software engineering tool landscape, benefiting various stakeholders, including industry, individual practitioners, and society. The article then delineates a roadmap for future research, outlining specific research paths and potential solutions for the research community to pursue. While not intended to be a definitive guide, the article aims to inspire further progress, with the ultimate goal of establishing efficient and green LLM4SE as a central element in the future of software engineering."
+W7140887005,,10.1145/3802827,2026,Enhancing Automated Unit Test Generation with Large Language Models: A Systematic Literature Review,Junwei Zhang,5,ACM Transactions on Software Engineering and Methodology,4,"Automated unit test generation is a fundamental yet challenging task in software engineering (SE), playing a critical role in ensuring software correctness, reliability, and maintainability. While traditional approaches such as search-based software testing and symbolic execution have achieved notable success, they often suffer from limited semantic understanding, high configuration costs, and scalability constraints. Recent advances in Large Language Models (LLMs) have fundamentally reshaped the landscape of automated unit testing by enabling models to reason over source code semantics and generate executable, context-aware test cases. Despite the rapid growth of this research area, a comprehensive and task-oriented synthesis of existing work remains lacking. This article presents a systematic literature review of LLM-based unit test generation. This review draws on research from leading SE and AI conferences and journals, including 69 papers published across 25 distinct venues, along with 47 high-quality preprint papers, bringing the total to 116. Our review aims to answer three key research questions: (1) which unit testing tasks have been addressed using LLMs, (2) how LLMs are adapted and integrated into the unit test generation pipeline, and (3) what datasets, benchmarks, and evaluation practices are employed in existing studies. To this end, we organize the literature from a task-centric perspective, covering test generation, test input generation, test oracle generation, and test evolution, and from a methodological perspective, categorizing LLM adaptation strategies into fine-tuning, prompt engineering, and agent-based approaches. Our analysis reveals that current research predominantly focuses on function- and class-level test generation, with comparatively limited attention given to test input generation, oracle construction, and long-term test evolution. Decoder-only LLMs, particularly GPT-family and LLaMA-based models, dominate the field, while encoder-only and encoder–decoder models remain underexplored. We further observe substantial disparities in dataset characteristics, programming language coverage, and evaluation metrics, which hinder fair comparison and reproducibility across studies. Based on empirical evidence extracted from the surveyed literature, we identify key challenges facing LLM-based unit test generation. Building on these findings, we outline several promising research directions, such as dataset optimization, structure-aware context modeling, agent coordination mechanisms, and benchmark enhancement. This review provides a consolidated and evidence-driven foundation for future research, aiming to advance the development of scalable, reliable, and practically applicable LLM-driven unit testing techniques."
+W4405543707,,10.1145/3708522,2024,Large Language Model for Vulnerability Detection and Repair: Literature Review and the Road Ahead,Xin Zhou,4,ACM Transactions on Software Engineering and Methodology,137,"The significant advancements in Large Language Models (LLMs) have resulted in their widespread adoption across various tasks within Software Engineering (SE), including vulnerability detection and repair. Numerous studies have investigated the application of LLMs to enhance vulnerability detection and repair tasks. Despite the increasing research interest, there is currently no existing survey that focuses on the utilization of LLMs for vulnerability detection and repair. In this paper, we aim to bridge this gap by offering a systematic literature review of approaches aimed at improving vulnerability detection and repair through the utilization of LLMs. The review encompasses research work from leading SE, AI, and Security conferences and journals, encompassing 43 papers published across 25 distinct venues, along with 15 high-quality preprint papers, bringing the total to 58 papers. By answering three key research questions, we aim to (1) summarize the LLMs employed in the relevant literature, (2) categorize various LLM adaptation techniques in vulnerability detection, and (3) classify various LLM adaptation techniques in vulnerability repair. Based on our findings, we have identified a series of limitations of existing studies. Additionally, we have outlined a roadmap highlighting potential opportunities that we believe are pertinent and crucial for future research endeavors."
+W7133294268,10.48550/arxiv.2603.00989,10.48550/arxiv.2603.00989,2026,Energy-Efficient Code Generation Using Large Language Models: A Systematic Literature Review,Sabiya Banu Masthan Ali,5,arXiv (Cornell University),0,"Large Language Models (LLMs) are increasingly used in software engineering for code generation, completion, translation, and repair. While substantial research has examined the energy consumption and environmental impact of LLM training and inference, considerably less attention has been given to the energy efficiency of the code produced by these models. Existing studies are fragmented across different models, programming tasks, benchmarks, hardware platforms, and measurement methods, making it difficult to draw consistent conclusions about the energy efficiency of LLM-generated code. No systematic synthesis currently consolidates these findings or identifies common trends, gaps, and methodological inconsistencies. To address this gap, we conduct a systematic literature review of existing empirical studies on the energy efficiency of LLM-generated code. We examine the extent to which LLMs generate energy-efficient code, the metrics used for evaluation, the benchmarks, datasets, and measurement tools employed, and the influence of prompting and fine-tuning strategies. Our findings show that the current evidence remains limited and highly heterogeneous. Most studies rely on conventional software-efficiency metrics and existing programming or code-generation benchmarks rather than evaluation methods specifically designed for LLM-generated code. The reported energy-efficiency outcomes are strongly dependent on the evaluated model, task, benchmark, and experimental configuration, and no single prompting strategy consistently improves energy efficiency across settings. We further find limited investigation of small language models, energy-aware fine-tuning, diverse programming languages, real-world software domains, and direct hardware-based energy measurement. Overall, the review highlights the lack of standardized evaluation practices and dedicated energy-efficiency benchmarks."
+W7133570722,,,2026,Energy-Efficient Code Generation Using Large Language Models: A Systematic Literature Review,Sabiya Banu Masthan Ali,5,arXiv (Cornell University),0,"Large Language Models (LLMs) are increasingly used in software engineering for code generation, completion, translation, and repair. While substantial research has examined the energy consumption and environmental impact of LLM training and inference, considerably less attention has been given to the energy efficiency of the code produced by these models. Existing studies are fragmented across different models, programming tasks, benchmarks, hardware platforms, and measurement methods, making it difficult to draw consistent conclusions about the energy efficiency of LLM-generated code. No systematic synthesis currently consolidates these findings or identifies common trends, gaps, and methodological inconsistencies. To address this gap, we conduct a systematic literature review of existing empirical studies on the energy efficiency of LLM-generated code. We examine the extent to which LLMs generate energy-efficient code, the metrics used for evaluation, the benchmarks, datasets, and measurement tools employed, and the influence of prompting and fine-tuning strategies. Our findings show that the current evidence remains limited and highly heterogeneous. Most studies rely on conventional software-efficiency metrics and existing programming or code-generation benchmarks rather than evaluation methods specifically designed for LLM-generated code. The reported energy-efficiency outcomes are strongly dependent on the evaluated model, task, benchmark, and experimental configuration, and no single prompting strategy consistently improves energy efficiency across settings. We further find limited investigation of small language models, energy-aware fine-tuning, diverse programming languages, real-world software domains, and direct hardware-based energy measurement. Overall, the review highlights the lack of standardized evaluation practices and dedicated energy-efficiency benchmarks."
+W7214983381,,10.1016/j.jss.2026.113129,2026,Large language models in software project management: A two-phase systematic literature review,Muhammad Saad Amin,4,Journal of Systems and Software,0,"Background Large Language Models (LLMs) are increasingly integrated into Software Engineering (SE) processes and practices. However, their application within Software Project Management (SPM) remains fragmented and insufficiently synthesized. This absence of systematic evidence prevents both researchers and practitioners from understanding where LLMs can meaningfully contribute to project management processes in relation to established standards such as the Project Management Body of Knowledge (PMBOK). Objectives This study aims to systematically synthesize existing evidence on the application and integration of LLMs in SPM by means of a two-phase method where Phase I systematically searches for existing secondary studies in the topic of interest, and Phase II (whenever applicable) carries out an SLR in the topic of interest. Method We conducted a two-phase method: Phase I is a tertiary study that used 14 research questions to assess the existence, coverage, and quality of prior secondary studies in LLMs applied to SPM. As no suitable SLR was identified from Phase I, we also conducted Phase II, which employed the SEGRESS guidelines to analyze primary studies that empirically evaluated the use of LLMs in SPM via five research questions. Phase II also assessed coverage across PMBOK processes and synthesized reported mechanisms, outcomes, research designs and evaluation approaches. Results The SLR (Phase II) shows that studies employ encoder-only and decoder-only LLM architectures in equal proportion (17 studies each). Their applications focus on a narrow set of SPM tasks, including effort estimation, project planning, and requirements-related activities. Coverage across PMBOK processes is uneven, with limited attention to areas such as procurement, stakeholder, quality, and cost management. Most studies rely on proof-of-concept evaluations, with scarce industrial validation and heterogeneous evaluation metrics. Conclusion The results indicate that current research on LLMs in SPM provides limited guidance for systematic adoption. One of the main drawbacks for such lack of guidance is the weak empirical validation in many of the included primary studies, limited PMBOK coverage, and lack of context-aware integration of LLMs in SPM. 1"
+W7166589016,,10.63317/5dnbkz4d44kz,2026,Automatic Generation of Discharge Summaries Using Large Language Models: A Systematic Literature Review,Lucas Molino-Piñar,3,Language Resources and Evaluation Conference (LREC),0,"Discharge summaries are critical documents for continuity of care, yet their manual creation imposes significant burdens on clinical staff.This systematic literature review examines current approaches to automatic generation of discharge summaries using Natural Language Processing (NLP) and Large Language Models (LLMs).Following the Kitchenham guidelines for systematic reviews in software engineering, we searched Scopus and PubMed databases for studies published between 2023 and 2026, identifying 9 primary studies from an initial pool of 102 papers.Our analysis reveals that GPT-4 and its variants dominate current research (appearing in 6 of 9 studies), while open-source alternatives like LLaMA show promise for privacy-preserving deployments.Evaluation primarily relies on automatic metrics (ROUGE, BLEU) combined with human expert assessment.Key challenges include hallucination rates ranging from 33% to 64%, information omission, integration with Electronic Health Record (EHR) systems, and context window limitations.Studies addressing factuality employ human-in-the-loop validation, prompt engineering techniques, and knowledge graph-based correction mechanisms.Despite these challenges, recent implementations demonstrate clinical feasibility, with one study achieving a 94.35% System Usability Score.This review provides a comprehensive synthesis of the state-of-the-art and identifies opportunities for future research in this rapidly evolving field."
+W7125135101,,10.1145/3747588,2025,A Survey on Large Language Models for Code Generation,Juyong Jiang,5,ACM Transactions on Software Engineering and Methodology,248,"Large Language Models (LLMs) have garnered remarkable advancements across diverse code-related tasks, known as Code LLMs, particularly in code generation that generates source code with LLM from natural language descriptions. This burgeoning field has captured significant interest from both academic researchers and industry professionals due to its practical significance in software development, e.g., GitHub Copilot . Despite the active exploration of LLMs for a variety of code tasks, either from the perspective of Natural Language Processing (NLP) or Software Engineering (SE) or both, there is a noticeable absence of a comprehensive and up-to-date literature review dedicated to LLM for code generation. In this survey, we aim to bridge this gap by providing a systematic literature review that serves as a valuable reference for researchers investigating the cutting-edge progress in LLMs for code generation. We introduce a taxonomy to categorize and discuss the recent developments in LLMs for code generation, covering aspects such as data curation, latest advances, performance evaluation, ethical implications, environmental impact, and real-world applications. In addition, we present a historical overview of the evolution of LLMs for code generation and provide a quantitative and qualitative comparative analysis of experimental results of code LLMs, sourced from their original papers to ensure a fair comparison on the HumanEval, MBPP, and BigCodeBench benchmarks, across various levels of difficulty and types of programming tasks, to highlight the progressive enhancements in LLM capabilities for code generation. We identify critical challenges and promising opportunities regarding the gap between academia and practical development. Furthermore, we have established a dedicated resource GitHub page ( https://github.com/juyongjiang/CodeLLMSurvey ) to continuously document and disseminate the most recent advances in the field."
+W7165734347,,10.21203/rs.3.rs-10126259/v1,2026,Are Large Language Models Ready for Quantum Software Engineering? A Multivocal Literature Review,Antonio Della Porta,4,Research Square,0,
+W7130568179,,10.1109/fllm67465.2025.11390990,2025,Towards the Integration of Large Language Models into the Software Development Life Cycle: A Systematic Literature Review,Mohammadamin Madani,6,International Conference on Foundation and Large Language Models (FLLM),1,"Models (LLMs) are reshaping the Software Development Life Cycle (SDLC) by introducing new levels of automation and intelligent assistance across core phases, including requirements engineering, design, testing, and maintenance. Previous works have examined the adoption of LLM in these areas, but have overlooked its influence on traditional practices in SDLC. Therefore, this paper builds upon this foundation and investigates, from a phase-level perspective, how LLMs interact with the SDLC as a whole and whether their presence necessitates rethinking of conventional methodologies. To achieve this objective, this article reviews the state of the field using a systematic literature review method. Peer-reviewed studies published between 2019 and 2025, examining the integration, benefits, and limitations of LLM within SDLC are analyzed. The findings indicate that LLMs are most commonly applied in implementation and testing, where they improve productivity through code generation, test automation, and fault localisation. However, their adoption also introduces challenges related to model hallucination, contextual misalignment, computational overhead, and ethical risks such as bias and opacity. Therefore, we synthesise several LLM integration patterns and stress various risk mitigation strategies that can be adopted to enhance the integration of LLMs into software engineering practices. The study concludes by outlining practical implications for software teams and proposing future research directions to improve LLM’s integration with agile, DevOps and CI/CD workflows."
+W7132830204,,10.6084/m9.figshare.30042556,2026,Are Large Language Models Ready for Quantum Software Engineering? A Multivocal Literature Review,Antonio Della Porta,1,Figshare,0,"This repository contains all the articles screened and included in our multivocal literature review on the use of Large Language Models in Quantum Software Engineering. The data stored in the repository was collected through queries conducted on ACM Digital Library, IEEE Xplore, Scopus, arXiv, and Google Search. The repository also reports the inclusion, exclusion, and quality-assessment criteria applied throughout the study selection process."
+W4394653832,10.48550/arxiv.2404.04566,10.48550/arxiv.2404.04566,2024,"Efficient and Green Large Language Models for Software Engineering: Literature Review, Vision, and the Road Ahead",Jieke Shi,3,arXiv (Cornell University),4,"Large Language Models (LLMs) have recently shown remarkable capabilities in various software engineering tasks, spurring the rapid growth of the Large Language Models for Software Engineering (LLM4SE) area. However, limited attention has been paid to developing efficient LLM4SE techniques that demand minimal computational cost, time, and memory resources, as well as green LLM4SE solutions that reduce energy consumption, water usage, and carbon emissions. This paper aims to redirect the focus of the research community towards the efficiency and greenness of LLM4SE, while also sharing potential research directions to achieve this goal. It commences with a brief overview of the significance of LLM4SE and highlights the need for efficient and green LLM4SE solutions. Subsequently, the paper presents a vision for a future where efficient and green LLM4SE revolutionizes the LLM-based software engineering tool landscape, benefiting various stakeholders, including industry, individual practitioners, and society. The paper then delineates a roadmap for future research, outlining specific research paths and potential solutions for the research community to pursue. While not intended to be a definitive guide, the paper aims to inspire further progress, with the ultimate goal of establishing efficient and green LLM4SE as a central element in the future of software engineering."
+W4401879229,,10.1109/compsac61105.2024.00096,2024,Systematic Literature Review of Prompt Engineering Patterns in Software Engineering,Yuya Sasaki,6,IEEE Annual International Computer Software and Applications Conference (COMPSAC),23,"Advancements in large language models (LLMs) are transforming software engineering through innovative prompt engineering strategies. By analyzing prompt-driven enhancements across key software engineering tasks, we present a sys-tematic literature review and a pioneering taxonomy elucidating the practical applications of prompt engineering in software engineering. Our taxonomy offers a foundational framework that clarifies the roles of prompt engineering and measures its impact, thereby guiding evolving AI -driven software engineering research and practices."
+W7159734530,,,2026,LLM-Assisted Empirical Software Engineering: Systematic Literature Review and Research Agenda,Victória Gomes,5,arXiv (Cornell University),0,"Context: Empirical Software Engineering (ESE) faces increasing challenges due to data scale, methodological complexity, and reproducibility concerns. Large Language Models (LLMs) have emerged as promising tools to support empirical workflows, yet their use remains fragmented, with no comprehensive synthesis to guide responsible adoption. Aims: This study analyzes how LLMs are used in ESE, examining supported tasks, phases of the empirical lifecycle, integration into workflows, reported benefits and limitations, and the extent of reproducibility-related reporting. It also identifies gaps and future research directions. Method: We conducted a systematic literature review of peer-reviewed papers (2020-2025) across 12 leading software engineering venues, resulting in 50 primary studies analyzed through qualitative and quantitative synthesis. Results: We identified 69 LLM-assisted tasks, mainly in mining software repositories and controlled experiments, focusing on classification, filtering, and evaluation. LLMs are used across multiple phases but are concentrated in data processing and analysis. Their integration is largely automation-oriented, with limited decision-support use. Benefits emphasize efficiency and scalability, while limitations include hallucinations, inconsistency, prompt sensitivity, and reproducibility issues. Reporting practices are often incomplete. Conclusion: LLM use in ESE is growing but remains automation-driven, with gaps in human-centered integration and transparency. We outline implications and research agenda for responsible use."
+W7131521417,,,2025,"Efficient and green large language models for software engineering: Literature review, vision, and the road ahead",Jieke Shi,3,Singapore Management University Institutional Knowledge (InK) (Singapore Management University),0,Singapore National Research Foundation (NRF)
+W7158894149,10.48550/arxiv.2604.26192,10.48550/arxiv.2604.26192,2026,LLM-Assisted Empirical Software Engineering: Systematic Literature Review and Research Agenda,Victória Gomes,5,arXiv (Cornell University),0,"Context: Empirical Software Engineering (ESE) faces increasing challenges due to data scale, methodological complexity, and reproducibility concerns. Large Language Models (LLMs) have emerged as promising tools to support empirical workflows, yet their use remains fragmented, with no comprehensive synthesis to guide responsible adoption. Aims: This study analyzes how LLMs are used in ESE, examining supported tasks, phases of the empirical lifecycle, integration into workflows, reported benefits and limitations, and the extent of reproducibility-related reporting. It also identifies gaps and future research directions. Method: We conducted a systematic literature review of peer-reviewed papers (2020-2025) across 12 leading software engineering venues, resulting in 50 primary studies analyzed through qualitative and quantitative synthesis. Results: We identified 69 LLM-assisted tasks, mainly in mining software repositories and controlled experiments, focusing on classification, filtering, and evaluation. LLMs are used across multiple phases but are concentrated in data processing and analysis. Their integration is largely automation-oriented, with limited decision-support use. Benefits emphasize efficiency and scalability, while limitations include hallucinations, inconsistency, prompt sensitivity, and reproducibility issues. Reporting practices are often incomplete. Conclusion: LLM use in ESE is growing but remains automation-driven, with gaps in human-centered integration and transparency. We outline implications and research agenda for responsible use."
+W4412164231,,10.1109/wsese66602.2025.00010,2025,On the Difficulties of Conducting and Replicating Systematic Literature Reviews Studies Using LLMs in Software Engineering,Katia Romero Felizardo,7,IEEE/ACM International Workshop on Methodological Issues with Empirical Studies in Software Engineering (WSESE),6,"The Software Engineering (SE) community has adopted Systematic Literature Reviews (SLRs) to summarize the state-of-the-art in specific research topics. SLRs offer benefits such as synthesizing evidence from diverse studies to generate auditable results following a reproducible approach, and identifying research gaps for future exploration. However, the process is effort-intensive, prone to errors, and lays various challenges during their conduction. To overcome some of these issues, there is a growing belief that Large Language Models (LLMs) can support systematic literature reviews. While the literature has shown promising results in social sciences, more evidence of its accuracy is needed in technical fields like SE. In this context, studies and replications are essential in verifying the benefits and drawbacks of applying LLMs in systematic literature reviews. This paper discusses the difficulties in conducting and replicating studies that adopt LLMs to support systematic literature in SE. As an implication, we identified the challenges of adopting LLM in SLRs and offered a list of open issues for future research."
+W4403413467,,10.1145/3674805.3686666,2024,ChatGPT application in Systematic Literature Reviews in Software Engineering: an evaluation of its accuracy to support the selection activity,Katia Romero Felizardo,5,Proceedings of the ... ACM-IEEE International Symposium on Empirical Software Engineering and Measurement/International Symposium on Empirical Software Engineering and Measurement,28,"Context: The Systematic Literature Review (SLR) process involves searching, selecting, and synthesizing relevant literature on a specific research topic for evidence-based decision-making in Software Engineering (SE). Due to the time-consuming of the SLR process, tool support is essential. Gap: ChatGPT is a significant advancement in Natural Language Processing (NLP), and it can potentially accelerate time-consuming and propone-error activities, such as the selection activity of the SLR process. Therefore, having a tool to assist in the selection process appears beneficial, and we argue that ChatGPT can facilitate the analysis of extensive studies, saving time and effort. Objective: We aim to evaluate the accuracy (i.e., studies correctly classified) of using ChatGPT–4.0 in SLR in SE, particularly to support the first stage, based on the title, abstract, and keywords. Method: We assessed the accuracy of utilizing ChatGPT for selecting studies, the first stage, to be included in two SLRs (SLR1 and SLR2), in contrast to the conventional method of reading the title and abstract. Results: The accuracy of ChatGPT supporting the initial selection activity was 75.3% (SLR1 – 101 correct selections: 48 inclusions and 53 exclusions; 33 incorrect selections: 17 inclusions and 16 exclusions) and 86.1% (SLR2 – 386 correct selections: 113 inclusions and 273 exclusions; 62 incorrect selections: 27 inclusions and 35 exclusions). Conclusions: Our accuracy results indicate that it is not advisable to completely outsource the selection process to ChatGPT. However, it could be valuable as a support tool, aiding novice researchers or even experienced ones when they are in doubt."
+W7164174508,,10.5281/zenodo.20481599,2026,Dataset for Requirements Analysis in Software Engineering Using Large Language Models,Daniel Mongeote-Tlachy,3,Zenodo (CERN European Organization for Nuclear Research),0,"This dataset supports the transparency and reproducibility of the systematic literature review and procedures reported in the article ""Requirements Analysis in Software Engineering Using Large Language Models.""The folder ""01-Need for a SLR search results"" documents the initial automated search process conducted in the Scopus and Web of Science databases. It includes the retrieved records based on the filters applied during the preliminary review of related literature.The folder ""02-Automatic search results"" contains the subsequent search stages carried out across selected digital library databases (IEEE Xplore, ACM Digital Library, ScienceDirect, and Scopus). It includes exported bibliographic records organized by search strings, as well as the inclusion and exclusion steps applied.The folder ""03-Atlas.TI"" contains all the data used and generated by the cualitatite data manager software.The file ""Dataset.xlsx"" serves as the main dataset. It contains fully curated data from the entire review process, including metadata for the selected studies: titles, venues, publication year, AI techniques, LLMs used, prompting strategies, evaluation methods, reported benefits, reported limitations and research gaps identified. It also includes raw and normalized quality assessment scores, aligned with the Kitchenham's methodology for software engineering research.Together, these materials enable full reproducibility of the review and provide a structured foundation for future research in AI-assisted web accessibility."
+W7123360572,,10.1109/access.2026.3652325,2026,Methods and Techniques of Agentic Software Engineering: A Systematic Literature Review,Nesreen A. Otoum,2,IEEE Access,6,"The development of agentic software engineering (ASE) is shifting the software development process, specifically by adopting autonomous agents into the fundamental software development process, where autonomous agents using large language models (LLMs) and artificial intelligence (AI) make decisions, write code, and collaborate across the entire software development lifecycle. The systematic review is a summary of literature on the agentic software engineering methodologies, frameworks, and techniques published in 2022-2025. We identify and compare methods based on autonomous coding, multi-agent systems, iterative refinement, and human-agent collaboration, drawing on 61 high-quality studies. We find that LLM-based agents show substantial potential in automated software engineering and that recent work is exploring new trends in agent cooperation, reasoning systems, and tool integration. We recognize such key challenges as the coordination procedures, reliability, explainability, and the ethical issues. This review characterizes the emerging role of agentic software engineering in improving the productivity of developers, the quality of their code, and the automation of their processes, and it offers a structured roadmap for future research. With the rapid evolution of the field, agentic approaches will tend to transform the conception, creation, and maintenance of software and provide more autonomous, adaptive, and intelligent systems."
+W7111070211,,10.5753/jserd.2025.5097,2025,Natural Language Processing in Software Engineering: A Systematic Literature Review,Gabriel Nogueira Pacheco,5,Journal of Software Engineering Research and Development,0,"Context: Software engineering (SE) artifacts and documents, such as requirements specifications, user stories, test cases, and concepts of operations (ConOps), are typically written in natural language, making their manipulation challenging. Natural Language Processing (NLP) is a viable solution for managing these tasks. Objective: To conduct a systematic literature review to explore the current use of NLP in SE artifacts and tasks, supplemented by a tertiary study focusing on the emerging role of Large Language Models (LLMs) in software engineering research. Method: We searched digital libraries for relevant papers and applied inclusion and exclusion criteria to filter the primary studies. We then analyzed NLP techniques applied to SE documents and examined their usage in this context. Our research methodology followed Kitchenham and Charters' guidelines. Additionally, we conducted a tertiary study to synthesize findings from existing systematic literature reviews and surveys specifically addressing LLMs in software engineering. Results: We selected 60 primary studies to identify the most common methods for NLP pipelines, feature extraction, language models, and machine learning algorithms used in SE. We also assessed the purposes of these methods, their benefits for SE, their difficulty, and their contribution to SE advancement. The tertiary study revealed a rapid proliferation of LLM-focused research, with comprehensive reviews documenting exponential growth in publications and widespread adoption across diverse SE tasks. Conclusion: Requirements are the most frequently addressed artifacts using NLP techniques, with preprocessing and part-of-speech (POS) tagging being widely used. There is a notable increase in the use of large language models for various SE tasks, such as requirements elicitation, source code generation, bug fixing, and software testing. The tertiary study confirms that LLMs represent a pivotal shift in the research landscape, warranting dedicated investigation to understand their transformative impact on NLP applications in software engineering."
+W4415064809,,10.1016/j.jss.2025.112758,2025,A systematic literature review of software engineering research on Jupyter notebook,Md. Saeed Siddik,3,Journal of Systems and Software,3,"• This research provides the first comprehensive systematic literature review on software engineering research specifically targeting Jupyter notebooks, identifying 199 primary studies published up to September 2025 and categorizing them into 11 core software engineering topics. • This research reveals that a large portion of the studies have been published outside traditional software engineering venues, with Human-Computer Interaction conferences like ACM Conference on Human Factors in Computing Systems (CHI) being the top publishing venues, highlighting the interdisciplinary nature of Jupyter Notebook research. • This research identifies a reusability gap in existing research, showing that only 82 out of 199 studies offer usable replication packages, and most are hosted on GitHub instead of permanent repositories, which violates open science best practices. • This research identifies that notebook-specific solutions for software engineering issues such as testing, refactoring, and documentation are relatively underexplored. Future directions include resolving duplicated execution numbers, refactoring inter-notebook clones, and generating grouped documentation for coherent-code cells are future directions derived from our study. • This research proposes the integration of modern AI-based solutions into Jupyter notebooks to support various software engineering topics, including code search and code generation. Additionally, future research should leverage advanced AI techniques (e.g., large language models), to improve conversational AI-powered assistants for automated code generation by multi-step workflow automation in data science notebooks. • Although the paper exceeds the recommended length due to the inclusion of detailed tables, figures, and categorized analyses (covering 11 topics and 21 subtopics), we believe that this extended content is essential for clearly and completely reporting our findings. As the first systematic literature review in this domain, we have carefully structured the paper to ensure readability. We believe the length is justified by the value and breadth of this paper’s contributions. Context : Jupyter Notebook has emerged as a versatile tool that transforms how researchers, developers, and data scientists conduct and communicate their work. As the adoption of Jupyter notebooks continues to rise, so does the interest from the software engineering research community in improving the software engineering practices for Jupyter notebooks. Objective : The purpose of this study is to analyze trends, gaps, and methodologies used in software engineering research on Jupyter notebooks. Method : We selected 199 relevant publications up to September 2025, following established systematic literature review guidelines. We explored publication trends, categorized them based on software engineering topics, and reported findings based on those topics. Results : The most popular venues for publishing software engineering research on Jupyter notebooks are related to human-computer interaction instead of traditional software engineering venues. Researchers have addressed a wide range of software engineering topics on notebooks, such as code reuse, readability, and execution environment. Although reusability is one of the research topics for Jupyter notebooks, only 82 of the 199 studies can be reused based on their provided URLs. Additionally, most replication packages are not hosted on permanent repositories for long-term availability and adherence to open science principles. Conclusion : Solutions specific to notebooks for software engineering issues, including testing, refactoring, and documentation, are underexplored. Future research opportunities exist in automatic testing frameworks, refactoring clones between notebooks, and generating group documentation for coherent code cells."
+W7134020591,,,2026,Large Language Models In High-Level Software Testing : A Literature Review,Toni Grönberg,1,Tampere University Institutional Repository (Tampere University),0,"Large language models (LLMs) have rapidly become influential tools in software engineering, raising new possibilities for supporting software testing across multiple levels of abstraction. Early applications of LLMs have focused strongly on low-level, code-centric tasks such as unit test generation, code repair, and test oracle inference. In contrast, high-level testing activities—including system, integration, acceptance, security, and GUI testing—have received considerably less attention despite their central role in validating the behaviour, reliability, and quality of complex software systems. This thesis investigates how LLMs are currently utilised in high-level software testing and examines the extent to which emerging research addresses the challenges inherent to these tasks. The thesis begins by introducing the foundations of software testing, defining test levels relevant to high-level dynamic testing, and outlining the essential properties of modern LLMs that enable their use in testing workflows. It then examines the traditional use of LLMs in testing, highlighting the limitations of early prompt-based approaches that relied on narrow context windows, limited autonomy, and weak integration with execution environments. Building on this, the thesis explores the rise of LLM-based testing agents capable of iterative reasoning, tool use, and adaptive test generation within feedback-driven loops. These agentic approaches represent a major shift from static prompt usage toward more autonomous testing systems that can interact with complex software artefacts. The analysis of high-level application areas demonstrates that LLM-supported methods are emerging in security testing, system testing, GUI exploration, acceptance testing, requirements analysis, and migration testing. Across these domains, LLMs contribute by interpreting naturallanguage specifications, generating structured scenarios, analysing outputs, and navigating interfaces in ways that traditional scripted automation struggles to achieve. At the same time, recurring limitations—such as hallucination, reproducibility issues, integration challenges, and inconsistent evaluation practices—highlight that LLM-based testing remains an evolving area with significant room for maturation. The thesis concludes that while high-level software testing has only recently begun to benefit from LLM-supported techniques, the field is progressing rapidly through the development of agent-based systems and hybrid architectures. LLMs are not yet reliable enough to replace human expertise in high-level testing, but they increasingly serve as capable collaborators that can accelerate test design, extend automation into previously inaccessible tasks, and support exploratory analysis. These findings suggest that continued research, better evaluation standards, and domain-informed model grounding are essential for advancing the practical adoption of LLMs in high-level software testing."
+W7135074624,,10.1109/sami68106.2026.11420586,2026,Large Language Models in Software Documentation and Modeling: A Literature Review and Findings,Lukáš Radoský,2,IEEE World Symposium on Applied Machine Intelligence and Informatics (SAMI),1,"Generative artificial intelligence attracts significant attention, especially with the introduction of large language models. Its capabilities are being exploited to solve various software engineering tasks. Thanks to their ability to understand natural language and generate natural language responses, large language models are great for processing various software documentation artifacts. At the same time, large language models excel at understanding structured languages, having the potential for working with software programs and models. We conduct a literature review on the usage of large language models for software engineering tasks related to documentation and modeling. We analyze articles from four major venues in the area, organize them per tasks they solve, and provide an overview of used prompt techniques, metrics, approaches to human-based evaluation, and major datasets."
+W7171820009,10.48550/arxiv.2607.26100,10.48550/arxiv.2607.26100,2026,Large Language Models for Software Engineering Diagrams: A Systematic Review of UML and ER modelling,Mojdeh Rahmanian,3,arXiv (Cornell University),0,"Large language models (LLMs) are increasingly applied to diagram-based software and data modelling. Among various modelling notations, UML and entity-relationship (ER) diagrams are the most widely adopted for software modelling and data modelling, respectively. Recent literature has investigated various applications of LLMs in diagram modelling; however, their effectiveness and limitations have not been extensively discussed. This systematic literature review analyses 64 studies published between 2023 and 2025, examining diagram coverage, modelling tasks, technical approaches, evaluation practices, and limitations. Our findings reveal significant concentration patterns and gaps. UML-based software modelling strongly dominates, with class diagrams receiving the most attention whilst behavioural diagrams and data modelling remain underrepresented. Diagram construction from natural language is the primary focus, with limited work on transformation, quality assurance, and consistency checking. GPT-based models are heavily prevalent, raising concerns about reproducibility and vendor dependence. Evaluation practices are heterogeneous, employing diverse metrics and custom datasets with limited benchmark reuse and inconsistent reporting of robustness and statistical significance. Common limitations include semantic inaccuracies, hallucinated diagram elements, sensitivity to prompt formulation, and reproducibility constraints. This survey provides the first systematic synthesis of LLM-based diagram modelling research, highlighting needs for standardised benchmarks, stronger evaluation protocols, broader diagram coverage, and techniques for improving semantic reliability and multi-view consistency."
+W7172068650,,,2026,Large Language Models for Software Engineering Diagrams: A Systematic Review of UML and ER modelling,Mojdeh Rahmanian,3,arXiv (Cornell University),0,"Large language models (LLMs) are increasingly applied to diagram-based software and data modelling. Among various modelling notations, UML and entity-relationship (ER) diagrams are the most widely adopted for software modelling and data modelling, respectively. Recent literature has investigated various applications of LLMs in diagram modelling; however, their effectiveness and limitations have not been extensively discussed. This systematic literature review analyses 64 studies published between 2023 and 2025, examining diagram coverage, modelling tasks, technical approaches, evaluation practices, and limitations. Our findings reveal significant concentration patterns and gaps. UML-based software modelling strongly dominates, with class diagrams receiving the most attention whilst behavioural diagrams and data modelling remain underrepresented. Diagram construction from natural language is the primary focus, with limited work on transformation, quality assurance, and consistency checking. GPT-based models are heavily prevalent, raising concerns about reproducibility and vendor dependence. Evaluation practices are heterogeneous, employing diverse metrics and custom datasets with limited benchmark reuse and inconsistent reporting of robustness and statistical significance. Common limitations include semantic inaccuracies, hallucinated diagram elements, sensitivity to prompt formulation, and reproducibility constraints. This survey provides the first systematic synthesis of LLM-based diagram modelling research, highlighting needs for standardised benchmarks, stronger evaluation protocols, broader diagram coverage, and techniques for improving semantic reliability and multi-view consistency."
+W7128096177,,,2026,Large Language Models in Software Documentation and Modeling: A Literature Review and Findings,Lukáš Radoský,2,arXiv (Cornell University),0,"Generative artificial intelligence attracts significant attention, especially with the introduction of large language models. Its capabilities are being exploited to solve various software engineering tasks. Thanks to their ability to understand natural language and generate natural language responses, large language models are great for processing various software documentation artifacts. At the same time, large language models excel at understanding structured languages, having the potential for working with software programs and models. We conduct a literature review on the usage of large language models for software engineering tasks related to documentation and modeling. We analyze articles from four major venues in the area, organize them per tasks they solve, and provide an overview of used prompt techniques, metrics, approaches to human-based evaluation, and major datasets."
+W7128002728,10.48550/arxiv.2602.04938,10.48550/arxiv.2602.04938,2026,Large Language Models in Software Documentation and Modeling: A Literature Review and Findings,Lukáš Radoský,2,arXiv (Cornell University),0,"Generative artificial intelligence attracts significant attention, especially with the introduction of large language models. Its capabilities are being exploited to solve various software engineering tasks. Thanks to their ability to understand natural language and generate natural language responses, large language models are great for processing various software documentation artifacts. At the same time, large language models excel at understanding structured languages, having the potential for working with software programs and models. We conduct a literature review on the usage of large language models for software engineering tasks related to documentation and modeling. We analyze articles from four major venues in the area, organize them per tasks they solve, and provide an overview of used prompt techniques, metrics, approaches to human-based evaluation, and major datasets."
+W4416231638,,10.1145/3841636,2026,Large Language Model for Verilog Code Generation: Literature Review and the Road Ahead,Guang Yang,17,ACM Computing Surveys,2,"Code generation represents a critical intersection of Software Engineering (SE) and Artificial Intelligence (AI). Within this broader landscape, Verilog, as a representative hardware description language (HDL), is fundamental to Electronic Design Automation (EDA), recent research has increasingly focused on leveraging Large Language Models (LLMs) to automate Verilog code generation, particularly at the Register Transfer Level (RTL) design. Despite growing interest, a comprehensive survey of this domain remains absent. This review addresses this gap by providing a systematic literature review of LLM-based Verilog code generation, analyzing 102 papers (70 published and 32 high-quality preprints) from SE, AI, and EDA venues. We structure our analysis around four key research questions: (1) identifying the LLMs utilized, (2) examining evaluation datasets and metrics, (3) categorizing generation techniques, and (4) analyzing alignment approaches. Furthermore, we synthesize findings to identify critical limitations in current studies regarding effectiveness and integration. Finally, we outline a roadmap highlighting potential opportunities for future research in LLM-assisted hardware design."
+W4416452806,10.48550/arxiv.2505.16697,10.48550/arxiv.2505.16697,2025,Software Architecture Meets LLMs: A Systematic Literature Review,Larissa Schmid,8,arXiv (Cornell University),4,"Large Language Models (LLMs) are used for many different software engineering tasks. In software architecture, they have been applied to tasks such as classification of design decisions, detection of design patterns, and generation of software architecture design from requirements. However, there is little overview on how well they work, what challenges exist, and what open problems remain. In this paper, we present a systematic literature review on the use of LLMs in software architecture. We analyze 18 research articles to answer five research questions, such as which software architecture tasks LLMs are used for, how much automation they provide, which models and techniques are used, and how these approaches are evaluated. Our findings show that while LLMs are increasingly applied to a variety of software architecture tasks and often outperform baselines, some areas, such as generating source code from architectural design, cloud-native computing and architecture, and checking conformance remain underexplored. Although current approaches mostly use simple prompting techniques, we identify a growing research interest in refining LLM-based approaches by integrating advanced techniques."
+W4393217463,,10.1007/978-981-97-1711-8_12,2024,A Pilot Study on AI-Assisted Code Generation with Large Language Models for Software Engineering,Hsiao-Chuan Liu,3,Communications in computer and information science,4,
+W4413788802,,10.3390/ai6090206,2025,Unlocking the Potential of the Prompt Engineering Paradigm in Software Engineering: A Systematic Literature Review,Irdina Wanda Syahputri,3,AI,8,"Prompt engineering (PE) has emerged as a transformative paradigm in software engineering (SE), leveraging large language models (LLMs) to support a wide range of SE tasks, including code generation, bug detection, and software traceability. This study conducts a systematic literature review (SLR) combined with a co-citation network analysis of 42 peer-reviewed journal articles to map key research themes, commonly applied PE methods, and evaluation metrics in the SE domain. The results reveal four prominent research clusters: manual prompt crafting, retrieval-augmented generation, chain-of-thought prompting, and automated prompt tuning. These approaches demonstrate notable progress, often matching or surpassing traditional fine-tuning methods in terms of adaptability and computational efficiency. Interdisciplinary collaboration among experts in AI, machine learning, and software engineering is identified as a key driver of innovation. However, several research gaps remain, including the absence of standardized evaluation protocols, sensitivity to prompt brittleness, and challenges in scalability across diverse SE applications. To address these issues, a modular prompt engineering framework is proposed, integrating human-in-the-loop design, automated prompt optimization, and version control mechanisms. Additionally, a conceptual pipeline is introduced to support domain adaptation and cross-domain generalization. Finally, a strategic research roadmap is presented, emphasizing future work on interpretability, fairness, and collaborative development platforms. This study offers a comprehensive foundation and practical insights to advance prompt engineering research tailored to the complex and evolving needs of software engineering."
+W4406325768,,10.1145/3712003,2025,"LLM-Based Multi-Agent Systems for Software Engineering: Literature Review, Vision, and the Road Ahead",Junda He,3,ACM Transactions on Software Engineering and Methodology,246,"Integrating Large Language Models (LLMs) into autonomous agents marks a significant shift in the research landscape by offering cognitive abilities that are competitive with human planning and reasoning. This article explores the transformative potential of integrating Large Language Models into Multi-Agent (LMA) systems for addressing complex challenges in software engineering (SE). By leveraging the collaborative and specialized abilities of multiple agents, LMA systems enable autonomous problem-solving, improve robustness, and provide scalable solutions for managing the complexity of real-world software projects. In this article, we conduct a systematic review of recent primary studies to map the current landscape of LMA applications across various stages of the software development lifecycle (SDLC). To illustrate current capabilities and limitations, we perform two case studies to demonstrate the effectiveness of state-of-the-art LMA frameworks. Additionally, we identify critical research gaps and propose a comprehensive research agenda focused on enhancing individual agent capabilities and optimizing agent synergy. Our work outlines a forward-looking vision for developing fully autonomous, scalable, and trustworthy LMA systems, laying the foundation for the evolution of Software Engineering 2.0."
+W7203792166,,10.1038/s41598-026-65738-4,2026,A maturity model for integrating large language models to address key challenges in software engineering education,Abdulrahman Alzahrani,2,Scientific Reports,0,"The integration of Large Language Models (LLM) into Software Engineering Education (SEE) holds significant potential to address key challenges such as diversity of students, rapid technological changes and resource constraints. This paper provides an extensive multivocal maturity model for the use of LLM in SEE, developed through multivocal literature review (MLR), an empirical survey and an interpretive structural modeling (ISM). The suggested model outlines and assesses the critical process areas (PAs) essential for incorporating LLM into SE curricula and provides a systematic way to improve educational practices. The study finds five maturity levels in SEE issues, such as Initial, Managed, Defined, Quantitatively Managed and Optimizing, related to student engagement, technological development, skills based on practical experiences and ethical issues. A mixed method approach (theoretical and practical) will be employed in exploring the potential of the LLMs for personalised learning, real-time feedback and interactive pedagogies. Teachers and institutions to determine their readiness level to implement LLM in SEE and continuously improve it for further innovation use the outcomes as a reference. The research is a contribution to the expanding research on how to leverage AI in the education and offers a novel framework to pursue the use of LLMs in the educational context of software engineering."
+W7116968648,,10.5335/rbca.v17i3.16310,2025,A Systematic Review about Large Language Models (LLMs) applied to Code Generation,Fabio Alecsandro Bacin,4,Revista Brasileira de Computação Aplicada,1,"Large Language Models (LLMs) for code generation represent a significant advancement in software development by boosting productivity, simplifying repetitive tasks, enabling automated testing, and promoting best practices. This paper presents a systematic literature review (SLR) of studies focused on LLMs applied to code generation. The review enhances understanding of the capabilities and limitations of these models, outlining both their benefits and challenges. The review protocol included a search on the Google Scholar database using relevant keywords related to LLMs and code generation. A total of 112 articles were initially retrieved, from which 15 were selected based on relevance and quality criteria. Of these, 8 were analyzed in depth to evaluate various approaches and outcomes, while the remaining 7 provided the theoretical foundation for the study. This work contributes to the growing body of knowledge in the field and supports future research and applications of LLMs in software engineering."
+W4414276043,,10.5753/ise.2025.14892,2025,Investigating the use of Large Language Models in software security requirements: Results of a literature review,Amália Melo,3,,1,"This study investigates the application of Large Language Models (LLMs) in the context of security requirements engineering through the conduction of a rapid literature review. The review enabled the characterization of current research in this domain with respect to: (i) the purposes for which LLMs are employed in security requirements activities; (ii) the families of LLMs explored (e.g., GPT, BERT, LLaMA), their capabilities (e.g., classification, generation), and underlying architectures (e.g., encoder, decoder, encoder-decoder); (iii) the techniques adopted for conditioning or guiding LLM behavior; (iv) the datasets used to train, fine-tune, or validate these models; and (v) the evaluation metrics applied to assess the performance of LLMs in supporting security requirements tasks. The findings contribute to a structured understanding of the current state of research and highlight key trends, gaps, and opportunities for advancing the use of LLMs in secure software engineering."
+W4405522167,,10.1109/conisoft63288.2024.00050,2024,State of the Art of the Security of Code Generated by LLMs: A Systematic Literature Review,Leonardo Criollo Ramírez,4,International Conference in Software Engineering Research and Innovation (CONISOFT),9,"AI -assisted programming has experienced a surge in popularity over the past few years, largely thanks to advancements in Large Language Model technologies. This has led to the emergence of tools like ChatGPT and GitHub Copilot. However, the use of AI models for code generation comes with a downside: the resulting code is susceptible to vulnerabilities, thus posing new challenges in the field of secure software development. In this study, we analyze the current state of research regarding the security of LLM generated code from the Software Engineering perspective. We conducted a Systematic Literature Review following the guidelines from Kitchenham et al. The search process included five sources: IEEE Xplore, ACM, Science Direct, Springer Link and Wiley Online Library. We also included an iteration of backward and forward snowballing. We obtained 3104 peer-reviewed studies though Quasi-Gold aided automated search and selected the most relevant ones through 5 stages. The final selection includes 15 primary studies from which we extracted and synthesized data. We identified seven different kinds of security vulnerability present in LLM generated code, six different mitigation strategies and practices, and four tools recommended by authors to use in conjunction with LLM code generation. Security related issues within LLM generated code have only just begun to be explored, initial research has already emphasized the significance of considering the inclusion of AI-powered code generation in software projects, as it carries the risk of introducing vulnerabilities at a higher rate than human-generated code. The vulnerabilities, practices and tools identified in this study, can potentially help developers to use LLM programming assistants more responsibly, making informed decisions when leveraging LLM technology."
+W4416418710,,10.1016/j.infsof.2025.107960,2025,Using LLMs to enhance code quality: A systematic literature review,Nawaf Alomari,4,Information and Software Technology,4,
+W7162402178,,10.20944/preprints202605.1638.v1,2026,A Systematic Literature Review on Generative AI in Software Engineering: Code Generation and Refactoring,Rafshan Bin Razzak,1,Preprints.org,0,"Generative AI has emerged as a transformative force in software engineering, particularly in code generation and refactoring, yet a comprehensive synthesis of its applications, challenges, and future directions remains lacking. This systematic literature review aims to consolidate existing research on generative AI in software engineering, focusing on its role in automating code generation, optimizing refactoring processes, and addressing broader implications such as ethics, security, and human-AI collaboration. We systematically analyze peer-reviewed studies to identify key trends, methodologies, and gaps in the literature, then categorize findings into dimensions including software development processes, testing, education, and interdisciplinary challenges. The review reveals that generative AI significantly accelerates code production and improves refactoring efficiency, but its adoption is constrained by issues such as code quality assurance, ethical concerns, and the need for effective human oversight. While advancements in large language models have expanded capabilities, their integration into real-world software engineering workflows requires further empirical validation. The study also highlights the growing importance of AI-assisted education and collaborative tools, suggesting that future research should prioritize transparency, robustness, and domain-specific adaptations. By synthesizing diverse perspectives, this review provides a foundational framework for researchers and practitioners to navigate the evolving landscape of generative AI in software engineering, while identifying critical areas for innovation and responsible deployment."
+W4406890462,,10.1109/access.2025.3535782,2025,Time-Series Large Language Models: A Systematic Review of State-of-the-Art,Shamsu Abdullahi,6,IEEE Access,41,"Large Language Models (LLMs) have transformed Natural Language Processing (NLP) and Software Engineering by fostering innovation, streamlining processes, and enabling data-driven decision-making. Recently, the adoption of LLMs in time-series analysis has catalyzed the emergence of time-series LLMs, a rapidly evolving research area. Existing reviews provide foundational insights into time-series LLMs but lack a comprehensive examination of recent advancements and do not adequately address critical challenges in this domain. This Systematic Literature Review (SLR) bridges these gaps by analysing state-of-the-art contributions in time-series LLMs, focusing on architectural innovations, tokenisation strategies, tasks, datasets, evaluation metrics, and unresolved challenges. Using a rigorous methodology based on PRISMA guidelines, over 700 studies from 2020 to 2024 were reviewed, with 59 relevant studies selected from journals, conferences, and workshops. Key findings reveal advancements in architectures and novel tokenization strategies tailored for temporal data. Forecasting dominates the identified tasks with 79.66% of the selected studies, while classification and anomaly detection remain underexplored. Furthermore, the analysis reveals a strong reliance on datasets from the energy and transportation domains, highlighting the need for more diverse datasets. Despite these advancements, significant challenges persist, including tokenization inefficiencies, prediction hallucinations, and difficulties in modelling long-term dependencies. These issues hinder the robustness, scalability, and adaptability of time-series LLMs across diverse applications. To address these challenges, this SLR outlines a research roadmap emphasizing the improvement of tokenization methods, the development of mechanisms for capturing long-term dependencies, the mitigation of hallucination effects, and the design of scalable, interpretable models for diverse time-series tasks."
+W7163994597,,10.46799/ajesh.v5i6.779,2026,Large Language Models for Intelligent Code Generation in Software Engineering: A Systematic Review and Future Research Directions,Ahmad Wali Noori,2,Asian Journal of Engineering Social and Health,0,"The proliferation of Large Language Models has catalyzed a paradigm shift in software engineering automation, yet existing literature reviews predominantly evaluate functional correctness while systematically neglecting security posture, maintainability, and multi-language deployment contexts. This study addresses the critical research gap regarding the absence of unified, multi-dimensional assessment protocols for LLM-generated code in production environments. Through a PRISMA-guided systematic review of 87 primary studies published between 2020 and 2025, this research examined architectural evolution, evaluation methodologies, and deployment barriers using a dual-phase qualitative and quantitative synthesis supported by a custom quality assessment framework. A five-criterion evaluation instrument was applied to ensure methodological rigor, with strong inter-rater reliability (Cohen’s kappa = 0.84). The findings reveal that decoder-only transformer architectures have achieved dominant performance on generative benchmarks, with Claude-3 Opus attaining 74.9% Pass@1 on HumanEval. However, only 12% of evaluated studies incorporated security metrics, fewer than 8% assessed maintainability, and benchmark contamination threatens the validity of reported generalization. The novelty of this work lies in proposing a unified evaluation framework that integrates functional correctness, Common Weakness Enumeration vulnerability scanning, and maintainability metrics, alongside a standardized multi-language protocol. The implications suggest that enterprise DevSecOps pipelines must embed static security analysis and cross-language quality gates before production integration. Future research should prioritize runtime verification, longitudinal productivity studies, and contamination-free benchmarks to ensure trustworthy deployment of generative AI in mission-critical software ecosystems."
+W4399431381,,10.1145/3643915.3644088,2024,Exploring the Potential of Large Language Models in Self-adaptive Systems,Jialong Li,6,International Symposium on Software Engineering for Adaptive and Self-Managing Systems (SEAMS),17,"Large Language Models (LLMs), with their abilities in knowledge acquisition and reasoning, can potentially enhance the various aspects of Self-adaptive Systems (SAS). Yet, the potential of LLMs in SAS remains largely unexplored and ambiguous, due to the lack of literature from flagship conferences or journals in the field, such as SEAMS and TAAS. The interdisciplinary nature of SAS suggests that drawing and integrating ideas from related fields, such as software engineering and autonomous agents, could unveil innovative research directions for LLMs within SAS. To this end, this paper reports the results of a literature review of studies in relevant fields, summarizes and classifies the studies relevant to SAS, and outlines their potential to specific aspects of SAS. Literature classification: www.github.com/545659928/LLM4SAS"
+W7214166792,,10.5281/zenodo.22802731,2026,Supplementary Material: Impostor Phenomenon and Generative AI in Software Engineering -- A Systematic Literature Review,Jonathan Felipe Hurtado Diaz,3,Zenodo (CERN European Organization for Nuclear Research),0,"FILE DESCRIPTIONS 00_protocol_slr.pdf Full SLR protocol (version 2.0, 2026-05-13) in Spanish. Includes research questions, PICOC+F framework, inclusion/exclusion criteria (CI1-CI6, CE1-CE7), screening procedure, MMAT quality assessment plan, data extraction strategy (28 fields), and synthesis approach. Based on Kitchenham & Charters (2007) and PRISMA 2020 (Page et al., 2021). Note: CI2 was expanded on 2026-06-06 (director approval) to include adjacent constructs (self-efficacy, cognitive load, psychological ownership, intrinsic motivation, stress). 00_protocol_slr_en.pdf English translation of the full SLR protocol (version 2.0). 01_screening_decisions_F2.xlsx Phase 2 (full-text) screening decisions for all 57 studies evaluated at full text. Includes inclusion/exclusion verdict with textual evidence cited from each paper (section, page). Dual reviewer (R1 + R2) with arbitration by thesis director (R3) for disagreements. 27 excluded with documented reasons; 9 not retrieved with access attempts documented. 02_data_extraction_matrix.xlsx Data extraction matrix (v2) with 28 fields across 8 categories for all 24 included publications (23 studies). Categories: metadata, study design, population/demographics, IP measurement, GenAI interaction, productivity/team impact, interventions, limitations. E0216 and E0235 are extracted separately (different constructs reported) but counted as one study. 03_mmat_quality_scores.xlsx Mixed Methods Appraisal Tool (MMAT 2018) quality profiles for all 23 included studies. Each study scored on 2 screening items + 5 design-specific items as Yes/No/NPD (no global score, per MMAT 2018 guidelines). Dual evaluation by R1 and R2; inter-rater agreement: kappa = 0.49 (moderate, N=145 items). E0224 excluded post-extraction (MMAT annulled). 04_snowballing_tracking.xlsx Forward and backward snowballing tracking (Wohlin, 2014) applied to 14 seed studies across 3 tiers. Two directed expansion passes via Semantic Scholar (May-July 2026). 36 unique candidates evaluated; 4 incorporated (E0401-E0404). Second pass yielded one additional study (E0404). The traversal was not exhaustive: only two of 14 planned seeds and two secondary seeds were traversed. The 25 exclusions made solely by R0 were validated by R1 (25/25 confirmed, 0 discrepancies). The notas_r1 column contains the justification for each validation. Additional sheet ""Validacion R0"" with summary by exclusion criterion. 05_screening_decisions_F1.xlsx Phase 1 (title + abstract) consolidated screening decisions for all 269 unique records after deduplication. Includes AI agent (R0) advisory classification and independent human reviewer decisions (R1, R2). R1 and R2 independently evaluated 61 records; 54 agreements (88.5%); binary kappa (Include+Dudoso vs Exclude) = 0.80. R0 coincided with the final human classification in 109/262 records (41.6%; kappa = 0.114); R0 classified 234/269 as ""Dudoso"" (87.0%). 06_screening_agent.py Python script for the LLM-based screening agent (Llama 3.3 70B via Groq API). Advisory role only (R0); human reviewers made all final inclusion/exclusion decisions independently. Classifies records against CI/CE criteria and assigns relevance scores. The agent classified 87% of records as ""Dudoso"" (uncertain); its recall for inclusion was 100% (zero false negatives). 07_search_strings_final.txt Complete search strings (v4.2) for all 6 databases (Scopus, Web of Science, IEEE Xplore, EBSCO/PsycINFO, SpringerLink, ScienceDirect). Two complementary sub-chains: Foundation-A (IP x SE population) and GenAI-B (IP + adjacent constructs x SE x GenAI). Includes database-specific syntax adaptations. 08_kappa_calculation.py Python script for computing Cohen's kappa inter-rater agreement between human reviewers (R1 and R2). Supports both F1 screening and MMAT quality assessment agreement calculations. 09_exclusions_extraction_phase.md Decisions made after F2 screening during full-text extraction: (1) E0224 excluded for not measuring the impostor phenomenon; (2) E0239 included as adjacent construct (relative self-efficacy); (3) E0216/E0235 treated as one study with two publications. Documents the rationale and PRISMA implications for each decision. 10_screening_agent_procedure.md Detailed procedure for the AI-assisted screening (section 4.3): how each reviewer used the screening agent outputs, independence protocol, and the separation between R0 (advisory) and R1/R2 (decision) roles. 11_exclusiones_snowballing.md Resolution of the 6 snowballing candidates that remained as ""Dudoso"" after R0 screening. All 6 were reviewed by R1 on 2026-09-15 and excluded: 4 failed CI2 (construct not in the taxative list) and 2 failed CE4 (non-empirical/WIP). Includes rationale per case and final snLine truncated
+W7133610361,,10.1145/3800580,2026,Novice Developers’ Perspectives on Adopting LLMs for Software Development: A Systematic Literature Review,Samuel Ferino,4,ACM Transactions on Software Engineering and Methodology,5,"Following the rise of large language models (LLMs), many studies have emerged in recent years focusing on exploring the adoption of LLM-based tools for software development by novice developers: computer science/software engineering students and early-career industry developers with two years or less of professional experience. These studies have sought to understand the perspectives of novice developers on using these tools, a critical aspect of the successful adoption of LLMs in software engineering. To systematically collect and summarise these studies, we conducted a systematic literature review (SLR) following the guidelines by Kitchenham et al. on 80 primary studies published between April 2022 and June 2025 to answer four research questions (RQs). In answering RQ1, we categorised the study motivations and methodological approaches. In RQ2, we identified the software development tasks for which novice developers use LLMs. In RQ3, we categorised the advantages, challenges, and recommendations discussed in the studies. Finally, we discuss the study limitations and future research needs suggested in the primary studies in answering RQ4. Throughout the paper, we also indicate directions for future work and implications for software engineering researchers, educators, and developers. Our research artifacts are publicly available at https://github.com/Samuellucas97/SupplementaryInfoPackage-SLR ."
+W4416545253,10.48550/arxiv.2503.07556,10.48550/arxiv.2503.07556,2025,Novice Developers' Perspectives on Adopting LLMs for Software Development: A Systematic Literature Review,Samuel Ferino,4,arXiv (Cornell University),1,"Following the rise of large language models (LLMs), many studies have emerged in recent years focusing on exploring the adoption of LLM-based tools for software development by novice developers: computer science/software engineering students and early-career industry developers with two years or less of professional experience. These studies have sought to understand the perspectives of novice developers on using these tools, a critical aspect of the successful adoption of LLMs in software engineering. To systematically collect and summarise these studies, we conducted a systematic literature review (SLR) following the guidelines by Kitchenham et al. on 80 primary studies published between April 2022 and June 2025 to answer four research questions (RQs). In answering RQ1, we categorised the study motivations and methodological approaches. In RQ2, we identified the software development tasks for which novice developers use LLMs. In RQ3, we categorised the advantages, challenges, and recommendations discussed in the studies. Finally, we discuss the study limitations and future research needs suggested in the primary studies in answering RQ4. Throughout the paper, we also indicate directions for future work and implications for software engineering researchers, educators, and developers. Our research artifacts are publicly available at https://github.com/Samuellucas97/SupplementaryInfoPackage-SLR."
+W7156306625,,10.22214/ijraset.2026.80859,2026,Analyzing Human-AI Collaborative Experience in Modern Software Development Using Large Language Models (Literature Review),Aman Singh,1,International Journal for Research in Applied Science and Engineering Technology,0,"The rise of Large Language Models which are often referred to as LLMs has brought about a significant change in how software development is conducted. This change marks the start of a collaborative relationship between humans and artificial intelligence. LLMs are not just automation tools anymore; they are increasingly seen as intelligent partners that help developers during various stages of the software development process. This literature review seeks to analyze critically the nature of collaboration between humans and AI in modern software engineering, focusing specifically on how LLMs impact developer experience, productivity, and decision-making. This paper draws on a wide array of academic studies, technical reports, and industry practices to look into the various roles that LLMs play in tasks like code generation, debugging, documentation, testing, and knowledge retrieval. It examines how developers engage with these systems in real-world situations, often participating in iterative and conversational workflows that mix human creativity with machine assistance. The review points out that LLMs not only speed up development timelines but also change cognitive workflows by lessening repetitive tasks and allowing developers to concentrate on more complex problem-solving and design issues. However, the use of LLMs in software development comes with its own set of challenges. This paper evaluates critically the concerns regarding the reliability and accuracy of outputs generated by these models, the danger of becoming too reliant on AI systems, and the possible decline of basic programming skills. It also discusses wider issues such as ethical concerns, data privacy, biases in the outputs from models, and the lack of transparency in decisions made by AI."
+W7147690639,,10.5281/zenodo.19368366,2026,LLMs on code smell detection and refactoring: a systematic literature review [Replication Package],Mateus Dias,2,Zenodo (CERN European Organization for Nuclear Research),0,Replication Package of research submitted to Journal of Software Engineering Research and Development
+W7130607038,,10.1145/3797276,2026,"LLM-as-a-Judge for Software Engineering: Literature Review, Vision, and the Road Ahead",Junda He,8,ACM Transactions on Software Engineering and Methodology,12,"The rapid integration of Large Language Models (LLMs) into software engineering (SE) has revolutionized tasks from code generation to program repair, producing a massive volume of software artifacts. This surge in automated creation has exposed a critical bottleneck: the lack of scalable and reliable methods to evaluate the quality of these outputs. Human evaluation, while effective, is very costly and time-consuming. Traditional automated metrics like BLEU rely on high-quality references and struggle to capture nuanced aspects of software quality, such as readability and usefulness. In response, the LLM-as-a-Judge paradigm, which employs LLMs for automated evaluation, has emerged. This approach leverages the advanced reasoning and coding capabilities of LLMs themselves to perform automated evaluations, offering a compelling path toward achieving both the nuance of human insight and the scalability of automated systems. Nevertheless, LLM-as-a-Judge research in the SE community is still in its early stages, with many breakthroughs needed. This forward-looking SE 2030 paper aims to steer the research community toward advancing LLM-as-a-Judge for evaluating LLM-generated software artifacts, while also sharing potential research paths to achieve this goal. We provide a literature review of existing SE studies on LLM-as-a-Judge and envision these frameworks as reliable, robust, and scalable human surrogates capable of evaluating software artifacts with consistent, multi-faceted assessments by 2030 and beyond. To validate this vision, we analyze the limitations of current studies, identify key research gaps, and outline a detailed roadmap to guide future developments of LLM-as-a-Judge in software engineering. While not intended to be a definitive guide, our work aims to foster further research and adoption of LLM-as-a-Judge frameworks within the SE community, ultimately improving the effectiveness and scalability of software artifact evaluation methods."
+W7162322444,,10.5281/zenodo.20387265,2026,(Artifact) Ai LitReview - Web-Based Tool for LLM-Assisted Systematic Literature Reviews,Anderson Deizepe,1,Zenodo (CERN European Organization for Nuclear Research),0,"AI LitReview is a collaborative web-based platform designed to support researchers throughout the entire Systematic Literature Review (SLR) process, from protocol definition to evidence synthesis. The tool integrates Large Language Models (LLMs) via the OpenRouter API (with access to over 365 models) and locally via Ollama, enabling AI-assisted study selection, structured data extraction, and RAG-based report generation. Key features include: configurable prompt templates with dynamic variables, multi-model screening comparison, full-text PDF screening, per-study cost and token tracking, human-in-the-loop decision-making, multi-user collaboration, Sankey-based Study Flow Diagram, and XLS export at each phase. This record contains the source code, demo video, and supplementary materials for the paper ""AI LitReview: A Web-Based Tool for AI-Assisted Systematic Literature Reviews"", submitted to the 40th Brazilian Symposium on Software Engineering (SBES 2026). The tool is publicly accessible at https://ailitreview.dzp.net.br and released under the MIT license."
+W4392781018,10.48550/arxiv.2403.07506,10.48550/arxiv.2403.07506,2024,"Robustness, Security, Privacy, Explainability, Efficiency, and Usability of Large Language Models for Code",Zhou Yang,5,arXiv (Cornell University),16,"Large language models for code (LLM4Code), which demonstrate strong performance (e.g., high accuracy) in processing source code, have significantly transformed software engineering. Many studies separately investigate the non-functional properties of LM4Code, but there is no systematic review of how these properties are evaluated and enhanced. This paper fills this gap by thoroughly examining 146 relevant studies, thereby presenting the first systematic literature review to identify seven important properties beyond accuracy, including robustness, security, privacy, explainability, efficiency, and usability. We discuss the current state-of-the-art methods and trends, identify gaps in existing research, and present promising directions for future study."
+W7125834859,,10.1109/tse.2026.3657900,2026,Agentic Recommender Systems: A Systematic Literature Review,Ivens Da Silva Portugal,3,IEEE Transactions on Software Engineering,2,"Recommender systems (RSs) are software systems that use machine learning techniques to suggest items, such as movies, products or routes to users based on input provided over time. These systems have been widely used by companies, such as Amazon, Google, and Netflix, who rely heavily on recommendations as a core element of their interaction with users. In the case of software engineering, RSs are used to recommend, for example, software tasks, developers, APIs, libraries, and bug fixes. More recently, advancements in natural language processing and large language models (LLMs) have led to the creation of LLM agents capable of generating plans, interacting with users and the environment, receiving feedback, and using tools. These developments have resulted in the use of LLM agents in RSs, giving rise to multi-agent RSs referred to as agentic recommender systems (ARSs). However, agentic RSs are not yet well characterized in terms of the basic elements of system modeling and design. This systematic literature review characterizes agentic recommender systems with respect to agents, roles, relationships, prompts, integration, use cases, strategies and evaluation methods. An analysis of published studies yields insights, such as (i) the identification of 13 distinct types of agents, (ii) the popularity of GPT models, (iii) a framework for ARSs, (iv) a generalized prompt, (v) the use of persona, cues, output format, and zero-shot prompting, (vi) the frequent use of Amazon, Yelp, and MovieLens datasets, (vii) the use of nDCG@k and Recall@k as common evaluation metrics, and (viii) the identification of many research gaps. This systematic literature review aims to support the continued research and development of agentic recommender systems."
+W7223032718,,10.5281/zenodo.23498699,2026,"Ambiguity in LLM-Based UML Generation: A Systematic Literature Review of Errors, Mitigation, and Validation",Driss Rami,5,Zenodo (CERN European Organization for Nuclear Research),0,"# Protocol and Replication Package for: Ambiguity in LLM-Based UML Generation: A Systematic Literature Review of Errors, Mitigation, and Validation This deposit contains the review protocol, the search strategies, the screening logs, the extracted dataset, the quality assessment scores, and the analysis scripts supporting the systematic literature review (SLR) reported in the manuscript: > Rami, D., El Bahri, A., Zougari, O., Kharmoum, N., & Ziti, S. *Ambiguity in LLM-Based UML Generation: A Systematic Literature Review of Errors, Mitigation, and Validation.* (2026). Manuscript under review. ## Review scope The review synthesizes 62 primary studies published between January 2023 and December 2025 on the use of large language models (LLMs) for transforming natural-language software requirements into Unified Modeling Language (UML) models. It addresses ten primary research questions (RQ1–RQ10) and four secondary review questions (SRQ1–SRQ4), covering: - requirement types transformed into UML models;- UML diagram types targeted;- LLM families, versions, and adaptation strategies;- validation mechanisms (syntactic, structural, semantic, traceability);- datasets and benchmarks;- evaluation metrics;- error and hallucination patterns;- reproducibility artifacts;- research gaps and a research agenda. ## Methodology The review follows the PRISMA 2020 statement (Page et al., 2021) and the Kitchenham & Charters (2007) guidelines for software engineering systematic reviews. Primary studies were retrieved from Scopus, Web of Science, IEEE Xplore, and the ACM Digital Library, using the database-specific queries documented in `search_strings/`. The full review protocol is provided in `protocol.pdf`. ## Contents of this deposit | File / folder | Description ||---|---|| `protocol.pdf` | Full review protocol: research questions, eligibility criteria (IC1–IC6, EC1–EC6), search strategy, screening procedure, data extraction form, quality assessment checklist (QA1–QA10), synthesis plan. || `search_strings/` | One text file per database (`scopus.txt`, `wos.txt`, `ieee.txt`, `acm_abstract.txt`, `acm_title.txt`) with the exact query, execution date, filters, and number of records retrieved. || `screening_logs/` | Title/abstract and full-text screening decisions with study identifiers and exclusion reasons (EC1–EC6). || `extraction_dataset.xlsx` | The 62 included primary studies coded across all fields of the extraction form (identification, requirement type, UML diagram type, LLM family/version, adaptation strategy, validation layer, metrics, error categories, reproducibility artifacts). || `quality_assessment.xlsx` | QA1–QA10 scores per included study, with total score and quality band (High / Moderate / Low). || `scripts/` | Python scripts used for deduplication (DOI, normalized title, fuzzy Jaccard matching ≥ 0.90), automatic keyword pre-screening, descriptive statistics, and figure generation. || `README.md` | This file. | ## Software environment - Python 3.11- pandas, matplotlib, rapidfuzz, openpyxl- Deduplication threshold: Jaccard similarity ≥ 0.90 on normalized titles- The scripts are provided as-is and were run on the dataset included in this deposit. ## How to cite If you use the protocol or the extraction dataset, please cite both the manuscript and this deposit: > Rami, D., El Bahri, A., Zougari, O., Kharmoum, N., & Ziti, S. (2026). *Protocol and replication package for: Ambiguity in LLM-Based UML Generation: A Systematic Literature Review*. Zenodo. https://doi.org/10.5281/zenodo.XXXXXXX ## License The protocol, dataset, and scripts are released under the Creative Commons Attribution 4.0 International license (CC BY 4.0). You are free to share and adapt the material for any purpose, provided that appropriate credit is given. ## Contact Driss Rami — drami.cpr@gmail.com — Intelligent Processing and Security of Systems (IPSS) Laboratory, Faculty of Sciences, Mohammed V University, Rabat, Morocco."
+W7114889968,,10.3390/bdcc9120320,2025,"A Systematic Literature Review of Retrieval-Augmented Generation: Techniques, Metrics, and Challenges",Andrew Brown,3,Big Data and Cognitive Computing,57,"Background: Retrieval-augmented generation (RAG) aims to reduce hallucinations and outdated knowledge by grounding LLM outputs in retrieved evidence, but empirical results are scattered across tasks, systems, and metrics, limiting cumulative insight. Objective: We aimed to synthesise empirical evidence on RAG effectiveness versus parametric-only baselines, map datasets/architectures/evaluation practices, and surface limitations and research gaps. Methods: This systematic review was conducted and reported in accordance with PRISMA 2020. We searched the ACM Digital Library, IEEE Xplore, Scopus, ScienceDirect, and DBLP; all sources were last searched on 13 May 2025. This included studies from January 2020–May 2025 that addressed RAG or similar retrieval-supported systems producing text output, met citation thresholds (≥15 for 2025; ≥30 for 2024 or earlier), and offered original contributions; excluded non-English items, irrelevant works, duplicates, and records without accessible full text. Bias was appraised with a brief checklist; screening used one reviewer with an independent check and discussion. LLM suggestions were advisory only; 2025 citation thresholds were adjusted to limit citation-lag. We used a descriptive approach to synthesise the results, organising studies by themes aligned to RQ1–RQ4 and reporting summary counts/frequencies; no meta-analysis was undertaken due to heterogeneity of designs and metrics. Results: We included 128 studies spanning knowledge-intensive tasks (35/128; 27.3%), open-domain QA (20/128; 15.6%), software engineering (13/128; 10.2%), and medical domains (11/128; 8.6%). Methods have shifted from DPR + seq2seq baselines to modular, policy-driven RAG with hybrid/structure-aware retrieval, uncertainty-triggered loops, memory, and emerging multimodality. Evaluation remains overlap-heavy (EM/F1), with increasing use of retrieval diagnostics (e.g., Recall@k, MRR@k), human judgements, and LLM-as-judge protocols. Efficiency and security (poisoning, leakage, jailbreaks) are growing concerns. Discussion: Evidence supports a shift to modular, policy-driven RAG, combining hybrid/structure-aware retrieval, uncertainty-aware control, memory, and multimodality, to improve grounding and efficiency. To advance from prototypes to dependable systems, we recommend: (i) holistic benchmarks pairing quality with cost/latency and safety, (ii) budget-aware retrieval/tool-use policies, and (iii) provenance-aware pipelines that expose uncertainty and deliver traceable evidence. We note the evidence base may be affected by citation-lag from the inclusion thresholds and by English-only, five-library coverage. Funding: Advanced Research and Engineering Centre. Registration: Not registered."
+W7155003632,10.48550/arxiv.2604.15385,10.48550/arxiv.2604.15385,2026,Prompt-Driven Code Summarization: A Systematic Literature Review,Afia Farjana,3,arXiv (Cornell University),0,"Software documentation is essential for program comprehension, developer onboarding, code review, and long-term maintenance. Yet producing quality documentation manually is time-consuming and frequently yields incomplete or inconsistent results. Large language models (LLMs) offer a promising solution by automatically generating natural language descriptions from source code, helping developers understand code more efficiently, facilitating maintenance, and supporting downstream activities such as defect localization and commit message generation. However, the effectiveness of LLMs in documentation tasks critically depends on how they are prompted. Properly structured instructions can substantially improve model performance, making prompt engineering-the design of input prompts to guide model behavior-a foundational technique in LLM-based software engineering. Approaches such as few-shot prompting, chain-of-thought reasoning, retrieval-augmented generation, and zero-shot learning show promise for code summarization, yet current research remains fragmented. There is limited understanding of which prompting strategies work best, for which models, and under what conditions. Moreover, evaluation practices vary widely, with most studies relying on overlap-based metrics that may not capture semantic quality. This systematic literature review consolidates existing evidence, categorizes prompting paradigms, examines their effectiveness, and identifies gaps to guide future research and practical adoption."
+W7155245368,,,2026,Prompt-Driven Code Summarization: A Systematic Literature Review,Afia Farjana,3,arXiv (Cornell University),0,"Software documentation is essential for program comprehension, developer onboarding, code review, and long-term maintenance. Yet producing quality documentation manually is time-consuming and frequently yields incomplete or inconsistent results. Large language models (LLMs) offer a promising solution by automatically generating natural language descriptions from source code, helping developers understand code more efficiently, facilitating maintenance, and supporting downstream activities such as defect localization and commit message generation. However, the effectiveness of LLMs in documentation tasks critically depends on how they are prompted. Properly structured instructions can substantially improve model performance, making prompt engineering-the design of input prompts to guide model behavior-a foundational technique in LLM-based software engineering. Approaches such as few-shot prompting, chain-of-thought reasoning, retrieval-augmented generation, and zero-shot learning show promise for code summarization, yet current research remains fragmented. There is limited understanding of which prompting strategies work best, for which models, and under what conditions. Moreover, evaluation practices vary widely, with most studies relying on overlap-based metrics that may not capture semantic quality. This systematic literature review consolidates existing evidence, categorizes prompting paradigms, examines their effectiveness, and identifies gaps to guide future research and practical adoption."
+W7112823045,,,2024,"Towards robust, secure, and privacy-aware large language models of code","YANG, Zhou",1,Singapore Management University Institutional Knowledge (InK) (Singapore Management University),0,"The field of software engineering has witnessed a surge in large language models specifically tailored to understand and process code, which we call large language models for code (LLM4Code). The increasing popularity of LLM4Code is inseparable from three key factors: the availability of extensive datasets compiled from diverse data sources, the advancements in deep learning algorithms and computational power that facilitate the training of these powerful models, and the active engagement and collaboration within the research community fostering innovation and the rapid exchange of ideas and methodologies. As evidenced by a series of studies, LLM4Code has been experiencing rapid development and achieving phenomenal success in various facets of software engineering. These powerful models have swiftly evolved from experimental prototypes to practical tools, integrating into the daily workflow of software developers around the globe. However, we unveil that LLM4Code fails to satisfy many non-functional requirements. For example, LLM4Code may not be robust, is vulnerable to data poisoning attacks, and may leak sensitive information. The dissertation starts with the first systematic literature review on the non-functional properties of LLM4Code, which provides a comprehensive understanding of the current status of LLM4Code. We identify six important non-functional properties of LLM4Code, including robustness, security, privacy, usability, explainability, and e ffi ciency. Then, this dissertation presents work on evaluating the robustness of LLM4Code. We highlight the naturalness requirements in operating adversarial attacks for LLM4Code. We propose using the masked language modeling capability of LLM4Code to generate adversarial perturbations that look natural to human developers. We design a two-step method to generate adversarial examples for code. Evaluation results show our method can generate more natural adversarial examples than existing methods, and our method can provide stronger robustness enhancement than baselines. The second work evaluates the security of LLM4Code, specifically the threats posed by backdoor attacks. In this work, we design a novel data poisoning method that can inject adaptive triggers into the training data of LLM4Code. The triggers are much stealthier than the baseline methods, capable of evading detection by both human developers and state-of-the-art defensive methods. Our evaluation results show that the stealthy backdoor attack can still achieve a high attack success rate. The third work unveils the memorization phenomena in LLM4Code. As part of the dissertation, we show that simple strategies can make models produce a vast amount of outputs that are exactly memorized from their training data. It is worrying that the memorized contents"
+W4410205944,,10.1145/3696673.3723053,2025,The Role of Generative AI Models in Requirements Engineering: A Systematic Literature Review,Poonkuzhali Vasudevan,2,ACM Southeast Conference (ACMSE),9,"The software engineering field is experiencing rapid growth, driven by recent advancements in Artificial Intelligence (AI), particularly in Generative AI (GenAI) and Large Language Models (LLMs). Requirements Engineering (RE), a critical phase in software development, involves gathering and defining software requirements. However, research on the impact of GenAI and LLMs within RE remains limited. This paper examines the adoption of GenAI in RE, with the aim of exploring its practical implications, identifying current research trends, and highlighting areas for future development. To achieve this, a systematic literature review was conducted, addressing three research questions and analyzing 44 studies published over the past decade. The findings reveal that GenAI models, especially LLMs, are extensively employed in a variety of RE tasks, underscoring the versatility and potential of LLMs in enhancing the RE process."
+W7160265065,,10.14569/ijacsa.2026.0170478,2026,The Impact of Modern AI on Software Development: A Systematic Literature Review,Meryem Bensaid,4,International Journal of Advanced Computer Science and Applications,0,"As large language models, and agentic AI systems are increasingly being integrated into software engineering, an expanding amount of empirical evidence surrounding these technologies has emerged. This systematic literature review examines the impact of modern AI techniques and tools in software development lifecycle phases and related activities, covering studies published between 2023 and 2025 and resulting in a corpus of 62 primary studies, investigating the role of large language models, AI agents and agentic AI workflows across lifecycle phases. The synthesis is guided by three research questions that address the entire software development cycle, reported impacts on development practices, outcomes, and constraints. This review fills a gap in the synthesis of modern AI applications and their impacts. It concludes that modern AI in software engineering is progressively evolving from a generation tool into a reasoning and coordination infrastructure layer, with ongoing efforts targeting the mitigation of identified limitations and the advancement of trustworthy agentic AI capabilities for dependable software engineering practices."
+W4415939046,10.48550/arxiv.2511.00776,10.48550/arxiv.2511.00776,2025,"A Systematic Literature Review of Code Hallucinations in LLMs: Characterization, Mitigation Methods, Challenges, and Future Directions for Reliable AI",Cuiyun Gao,8,arXiv (Cornell University),2,"Model hallucination is one of the most critical challenges faced by Large Language Models (LLMs), especially in high-stakes code intelligence tasks. As LLMs become increasingly integrated into software engineering tasks, understanding and mitigating hallucination in code becomes essential. In this survey, we provide a systematic review of hallucination phenomena in code-oriented LLMs from four key perspectives. First, we begin by surveying 60 papers to define hallucination in the context of code and summarize its primary causes, such as data noise, exposure bias, and insufficient semantic grounding, while also tracing recent trends in literature across natural language processing (NLP) and software engineering communities. Second, we review model hallucination surveys in a broader span and summarize representative hallucination mitigation strategies, such as knowledge-enhanced generation, constrained decoding, and post-editing. Third, we review approaches targeted for code intelligence and highlight code-specific challenges that aggravate hallucination, including syntax sensitivity, strict type systems, and dependence on external libraries. Meanwhile, we analyze how emerging code intelligence tasks, e.g., program analysis, symbolic execution, and unit testing, are utilized to detect and mitigate hallucinations. Fourth, we summarize current evaluation benchmarks, ranging from static metrics to dynamic checks, e.g., compilation and execution correctness, and emphasize the need for hallucination-oriented benchmarks."
+W7208824083,,10.66104/4hndmk68,2026,Artificial Intelligence Tools for Productivity Enhancement: A Systematic Literature Review,Marcelo Almeida Pereira Guedes,1,Nexus Science Review,1,"Artificial Intelligence (AI), particularly generative AI based on Large Language Models (LLMs), has rapidly transformed the execution of knowledge-intensive activities across multiple domains. AI-powered tools such as ChatGPT, GitHub Copilot, Microsoft Copilot, Google Gemini, Claude, and Notion AI have increasingly been adopted to automate repetitive tasks, support decision-making, accelerate software development, and improve content production. However, despite the rapid expansion of these technologies, scientific evidence regarding their effectiveness as productivity enhancers remains distributed across different research areas. This study presents a systematic literature review aimed at synthesizing current evidence on the role of AI tools in improving productivity in business, education, software engineering, and scientific research. A structured literature search was conducted across major academic databases, including Google Scholar, IEEE Xplore, ACM Digital Library, ScienceDirect, SpringerLink, and Scopus. Studies published between 2020 and 2026 were analyzed according to predefined inclusion and exclusion criteria. The reviewed literature indicates that AI tools can improve productivity by reducing task completion time, assisting knowledge creation, supporting programming activities, and enhancing information processing. Nevertheless, significant challenges remain, including inaccurate outputs, algorithmic bias, privacy concerns, ethical risks, and excessive reliance on automated systems. The findings suggest that AI achieves the greatest productivity benefits when used as a collaborative technology that augments human capabilities rather than replacing human expertise. Future research should investigate long-term productivity impacts, organizational adaptation strategies, responsible AI governance, and the integration of advanced multimodal AI systems into professional workflows."
+W4414155484,,10.1145/3766552,2025,Metamorphic Testing of Deep Code Models: A Systematic Literature Review,Ali Asgari,4,ACM Transactions on Software Engineering and Methodology,3,"Large language models and deep learning models designed for code intelligence have revolutionized the software engineering field due to their ability to perform various code-related tasks. These models can process source code and software artifacts with high accuracy in tasks such as code completion, defect detection, and code summarization; therefore, they can potentially become an integral part of modern software engineering practices. Despite these capabilities, robustness remains a critical quality attribute for deep-code models as they may produce different results under varied and adversarial conditions (e.g., variable renaming). Metamorphic testing has become a widely used approach to evaluate models’ robustness by applying semantic-preserving transformations to input programs and analyzing the stability of model outputs. While prior research has explored testing deep learning models, this systematic literature review focuses specifically on metamorphic testing for deep code models. By studying 45 primary papers, we analyze the transformations, techniques, and evaluation methods used to assess robustness. Our review summarizes the current landscape, identifying frequently evaluated models, programming tasks, datasets, target languages, and evaluation metrics, and highlights key challenges and future directions for advancing the field."
+W7126253154,10.48550/arxiv.2601.20879,10.48550/arxiv.2601.20879,2026,"A Survey on Large Language Model Impact on Software Evolvability and Maintainability: the Good, the Bad, the Ugly, and the Remedy",Bruno Claudino Matias,6,arXiv (Cornell University),0,"Context. Large Language Models (LLMs) are increasingly embedded in software engineering workflows for tasks including code generation, summarization, repair, and testing. Empirical studies report productivity gains, improved comprehension, and reduced cognitive load. However, evidence remains fragmented, and concerns persist about hallucinations, unstable outputs, methodological limitations, and emerging forms of technical debt. How these mixed effects shape long-term software maintainability and evolvability remains unclear. Objectives. This study systematically examines how LLMs influence the maintainability and evolvability of software systems. We identify which quality attributes are addressed in existing research, the positive impacts LLMs provide, the risks and weaknesses they introduce, and the mitigation strategies proposed in the literature. Method. We conducted a systematic literature review. Searches across ACM DL, IEEE Xplore, and Scopus (2020 to 2024) yielded 87 primary studies. Qualitative evidence was extracted through a calibrated multi-researcher process. Attributes were analyzed descriptively, while impacts, risks, weaknesses, and mitigation strategies were synthesized using a hybrid thematic approach supported by an LLM-assisted analysis tool with human-in-the-loop validation. Results. LLMs provide benefits such as improved analyzability, testability, code comprehension, debugging support, and automated repair. However, they also introduce risks, including hallucinated or incorrect outputs, brittleness to context, limited domain reasoning, unstable performance, and flaws in current evaluations, which threaten long-term evolvability. Conclusion. LLMs can strengthen maintainability and evolvability, but they also pose nontrivial risks to long-term sustainability. Responsible adoption requires safeguards, rigorous evaluation, and structured human oversight."
+W7163154377,,10.22214/ijraset.2026.83359,2026,Generative AI in Mobile Application Development: A Systematic Literature Review,Balram Kumar,2,International Journal for Research in Applied Science and Engineering Technology,0,"Generative Artificial Intelligence (GenAI) tools have become central to modern software engineering, fundamentally transforming how developers design, write, debug, test, and document code. This systematic literature review examines the measurable impact of GenAI tools — specifically GitHub Copilot, OpenAI GPT-4/ChatGPT, Google Gemini, Anthropic Claude 3, and Meta Code LLaMA — on Android and Flutter mobile application development. Following the PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) framework, we screened 120 candidate papers and identified 35 high-quality peer-reviewed publications from January 2020 to March 2026 across five major academic databases. The review addresses four research questions covering tool usage patterns, productivity gains, quality and security risks, and future research directions. Findings confirm mean productivity gains of 45–55% for code generation, 30–40% for debugging, 35–50% for UI/UX automation, and up to 60% for documentation writing. However, 25–40% of AI-generated code contains correctness issues and approximately 40% of security-critical AI code contains exploitable vulnerabilities. A novel Seven-Phase GenAI Integration Framework is proposed, mapping specific tools to each phase of the mobile development lifecycle. Six priority areas for future research are identified, including mobile-specific fine-tuned models, real-time RAG for live API documentation, and on-device privacy-preserving LLMs."
+W7165367028,,10.51582/interconf.19-20.05.2026.017,2026,Taxonomy of Prompt Injection Attacks and Analysis of Defense Mechanisms in Large Language Model-Based Chatbots,Telman Yusifov,3,InterConf,0,"Large Language Models have become critical infrastructure across healthcare, finance, and software engineering, yet their instruction-following design creates a structural class of security vulnerabilities known as prompt injection. This paper presents a comprehensive taxonomy of prompt injection attacks targeting LLM-based chatbots, organizes the threat landscape across seven categories covering both direct and indirect injection vectors, and evaluates defense mechanisms at three operational layers. The theoretical framework is validated through an empirical benchmark (PI-Bench) testing 210 prompt injection payloads across seven attack categories against five open-weight LLM models in a controlled local environment. Empirical findings demonstrate attack success rates ranging from 10.5% (LLaMA 3.1 8B) to 83.8% (Mistral 7B), confirming that alignment training quality is the primary determinant of injection resistance rather than model size. Across all models, Jailbreak/Roleplay attacks proved most effective at 34.7% aggregate ASR, while Gradient-Based attacks were least effective at 22.7%. The literature review documents attack success rates reaching 87.2% against state-of-the-art proprietary models and an 86.1% success rate for indirect injection against production applications. No single defensive measure provides adequate coverage, and a defense-in-depth architecture is necessary for robust protection."
+W7171902466,,,2026,Preliminary Guidelines for Using and Evaluating GenAI Tools to Support Systematic Literature Reviews,Barbara Kitchenham,6,arXiv (Cornell University),0,"Context: Generative AI (GenAI) and Large Language Models (LLMs) are increasingly used for academic tasks in software engineering and beyond, including systematic literature reviews (SLRs). However, while capable of summarizing text, there is no guarantee they can meet the rigour, reliability, and transparency that SLRs require. Objectives: To support researchers intending to conduct SLRs using GenAI or those conducting empirical studies evaluating how well GenAI supports SLR tasks. Methods: First, we conducted a rapid review to identify studies that propose guidelines for evaluating and using GenAI and LLMs to support SLRs. Second, we drew on thought experiments, relevant guidance from the literature, and our own experience conducting SLRs and evaluating tools to develop recommendations for how to use and assess GenAI in the context of SLRs. Results: We discuss the problems researchers face when evaluating GenAI for SLRs. We identify and explain process issues to consider when planning, conducting, and reporting both SLRs using GenAI and evaluations of GenAI tools. Finally, we summarize our results as a set of process recommendations, which we name GUEST (GenAI Use and Evaluation in SLR Tasks). Conclusion: We argue that GenAI requires human oversight and is not currently capable of unsupervised systematic studies. However, it offers the prospect of cost-effective assistance for some repetitive tasks and for additional validation of some complex tasks. Our GUEST recommendations should help software engineering researchers both to conduct and report trustworthy SLRs using GenAI and to provide rigorous independent evaluation studies."
+W7134063879,,10.17632/nf7dz42sfn,2026,Supplementary Material for A Conceptual Model for Born-Accessible Web Accessibility Remediation Using Large Language Models,Guillermo Vera-Amaro,2,Mendeley Data,0,"This dataset supports the transparency and reproducibility of the systematic literature review and procedures reported in the article ""A Conceptual Model for Born-Accessible Web Accessibility Remediation Using Large Language Models."" The folder ""01-Related Work search results"" documents the initial automated search process conducted in the Scopus and Web of Science databases. It includes the retrieved records based on the filters applied during the preliminary review of related literature. The folder ""02-Automatic search results"" contains the subsequent search stages carried out across selected digital library databases (Scopus, Web of Science, and arXiv). It includes exported bibliographic records organized by search strings, as well as the inclusion and exclusion steps applied. The file ""Dataset_Conceptual_model_Web_Accessibility.xlsx"" serves as the main dataset. It contains fully curated data from the entire review process, including metadata for the selected studies: titles, venues, publication year, AI techniques, LLMs used, prompting strategies, evaluation methods, and referenced accessibility standards. It also includes raw and normalized quality assessment scores, aligned with the Kitchenham's methodology for software engineering research. Also includes a link to the code repository for the validation phase of the model. Together, these materials enable full reproducibility of the review and provide a structured foundation for future research in AI-assisted web accessibility."
+W7128496016,,10.1145/3796522,2026,A Systematic Literature Review of Parameter-Efficient Fine-Tuning for Large Code Models,Saima Afrin,3,ACM Transactions on Software Engineering and Methodology,1,"The rise of Artificial Intelligence (AI)-and particularly Large Language Models (LLMs) for code–has reshaped Software Engineering (SE) by enabling the automation of tasks such as code generation, bug detection, and repair. However, these models require significant computational resources for training and fine-tuning, posing challenges for real-world adoption in resource-constrained environments. To address this, the research community has increasingly turned to Parameter-Efficient Fine-Tuning (PEFT)–a class of techniques that enables the adaptation of large models by updating only a small subset of parameters, rather than the entire model. In this Systematic Literature Review (SLR), we examine the growing application of PEFT techniques–across a wide range of software engineering tasks. We analyze how these methods are used to optimize various deep learning (DL) architectures, focusing on their impact on both performance and efficiency. Our study synthesizes findings from 28 peer-reviewed papers, identifying patterns in configuration strategies and adaptation trade-offs. The outcome of this review is a comprehensive taxonomy that categorizes PEFT usage by task type, distinguishing between generative (e.g., Code Summarization) and non-generative (e.g., Code Clone Detection) scenarios. Our findings aim to inform future research and guide the practical deployment of PEFT in sustainable, AI-powered software development. Our artifacts are publicly available at https://github.com/alvi75/SLR-PEFT"
+W7171690888,10.48550/arxiv.2607.24991,10.48550/arxiv.2607.24991,2026,Preliminary Guidelines for Using and Evaluating GenAI Tools to Support Systematic Literature Reviews,Barbara Kitchenham,6,arXiv (Cornell University),0,"Context: Generative AI (GenAI) and Large Language Models (LLMs) are increasingly used for academic tasks in software engineering and beyond, including systematic literature reviews (SLRs). However, while capable of summarizing text, there is no guarantee they can meet the rigour, reliability, and transparency that SLRs require. Objectives: To support researchers intending to conduct SLRs using GenAI or those conducting empirical studies evaluating how well GenAI supports SLR tasks. Methods: First, we conducted a rapid review to identify studies that propose guidelines for evaluating and using GenAI and LLMs to support SLRs. Second, we drew on thought experiments, relevant guidance from the literature, and our own experience conducting SLRs and evaluating tools to develop recommendations for how to use and assess GenAI in the context of SLRs. Results: We discuss the problems researchers face when evaluating GenAI for SLRs. We identify and explain process issues to consider when planning, conducting, and reporting both SLRs using GenAI and evaluations of GenAI tools. Finally, we summarize our results as a set of process recommendations, which we name GUEST (GenAI Use and Evaluation in SLR Tasks). Conclusion: We argue that GenAI requires human oversight and is not currently capable of unsupervised systematic studies. However, it offers the prospect of cost-effective assistance for some repetitive tasks and for additional validation of some complex tasks. Our GUEST recommendations should help software engineering researchers both to conduct and report trustworthy SLRs using GenAI and to provide rigorous independent evaluation studies."
+W4417523743,10.48550/arxiv.2504.21569,10.48550/arxiv.2504.21569,2025,A Systematic Literature Review of Parameter-Efficient Fine-Tuning for Large Code Models,Md. Zahidul Haque,3,arXiv (Cornell University),1,"The rise of Artificial Intelligence (AI)-and particularly Large Language Models (LLMs) for code-has reshaped Software Engineering (SE) by enabling the automation of tasks such as code generation, bug detection, and repair. However, these models require significant computational resources for training and fine-tuning, posing challenges for real-world adoption in resource-constrained environments. To address this, the research community has increasingly turned to Parameter-Efficient Fine-Tuning (PEFT)-a class of techniques that enables the adaptation of large models by updating only a small subset of parameters, rather than the entire model. In this Systematic Literature Review (SLR), we examine the growing application of PEFT techniques-across a wide range of software engineering tasks. We analyze how these methods are used to optimize various deep learning (DL) architectures, focusing on their impact on both performance and efficiency. Our study synthesizes findings from 27 peer-reviewed papers, identifying patterns in configuration strategies and adaptation trade-offs. The outcome of this review is a comprehensive taxonomy that categorizes PEFT usage by task type, distinguishing between generative (e.g., Code Summarization) and non-generative (e.g., Code Clone Detection) scenarios. Our findings aim to inform future research and guide the practical deployment of PEFT in sustainable, AI-powered software development. Our artifacts are publicly available at https://github.com/alvi75/SLR-PEFT"
+W4416642449,,10.1016/j.infsof.2025.107979,2025,Quality assessment of software requirements using artificial intelligence methods: A systematic literature review,Elise Wolf,3,Information and Software Technology,7,"Context: The quality of requirements specifications is a critical success factor in software development. Assuring high-quality requirements, specifically in an automated way, poses a significant challenge due to their unstructured and multi-modal character. With the rise of deep learning and large language models (LLMs), new opportunities have developed to assess the quality of requirements automatically, particularly user stories in the context of agile software engineering, where short development cycles require efficient tool support. Objective: This study aims to systematically review and investigate the current landscape of approaches based on artificial intelligence techniques such as natural language processing and deep learning for assessing the quality of software requirements. The investigation focuses on the artificial intelligence techniques adopted, quality aspects considered, datasets used to tune and evaluate the proposed approaches, and their performance. Method: We conducted a systematic literature review of 26 peer-reviewed papers published between 2019 and 2025. We selected the papers after a title and abstract review of 353 papers identified through a literature databases query and forward–backward snowballing. Results: The results reveal significant overlap among considered quality aspects, which can be mapped onto the higher-order requirements quality model INVEST. Most studies focus on assessing requirement quality rather than improving requirements and rely heavily on synthetic and public datasets. LLMs have rapidly gained popularity since 2023, though model evaluation strategies remain inconsistent. Metrics such as accuracy, precision, recall, and F1-Score are common, yet a few studies use semantic or expert-based evaluations. Conclusion: The field is evolving toward LLM-driven, semantically rich models, yet lacks methodological standardization, reproducible datasets for evaluating the models, and integration of the approaches with real-world requirements engineering processes. Future work should address these limitations by developing benchmark datasets, standardizing evaluation metrics, and exploring hybrid systems that combine AI-based and traditional requirements quality assurance approaches."
+W4412081931,,10.1109/tse.2025.3586411,2025,"Open Source, Hidden Costs: A Systematic Literature Review on OSS License Management",Boyuan Li,6,IEEE Transactions on Software Engineering,5,"Integrating third-party software components is a common practice in modern software development, offering significant advantages in terms of efficiency and innovation. However, this practice is fraught with risks related to software licensing. A lack of understanding may lead to disputes, which can pose serious legal and operational challenges. To these ends, both academia and industry have conducted various investigations and proposed solutions and tools to deal with these challenges. However, significant limitations still remain. Moreover, the rapid evolution of open-source software (OSS) licenses, as well as the rapidly incorporated generative software engineering techniques, such as large language models for code (CodeLLMs), are placing greater demands on the systematic management of software license risks. To unveil the severe challenges and explore possible future directions, we conduct the first systematic literature review (SLR) on 80 carefully selected OSS license-related papers, classifying existing research into three key categories, i.e., license identification, license risk assessment, and license risk mitigation. Based on these, we discuss challenges in existing solutions, conclude the opportunities to shed light on future research directions and offer practical recommendations for practitioners.We hope this thorough review will help bridge the gaps between academia and industry and accelerate the ecosystem-wide governance of legitimate software risks within the software engineering community."
+W7223205246,,10.17605/osf.io/cpzsa,2026,"Automated Adversary Emulation and Autonomous Penetration Testing: A Systematic Literature Review of Attack Graphs, Reinforcement Learning, LLM Agents, and Framework Validity",Lynet Tsitsi Svotwa,1,Open Science Framework,0,"This systematic literature review investigates the evolution of automated adversary emulation and autonomous penetration testing, from attack graphs and classical planning approaches to reinforcement learning and large language model (LLM) based autonomous agents. The review examines how these approaches model attacker objectives, represent target environments, select attack actions and evaluate performance. It also assesses empirical evidence concerning the validity and limitations of the Cyber Kill Chain and MITRE ATT&CK frameworks for representing adversary behaviour. The review will use structured academic searches, predefined eligibility criteria, transparent study selection, quality appraisal and comparative narrative synthesis. Reporting will be guided by PRISMA 2020 and PRISMA-S, with methodological consideration of established software-engineering systematic review guidance. Preliminary literature searches and paper retrieval have commenced. Formal screening, quality appraisal and synthesis are planned to follow the finalisation of this protocol"
+W7167225127,,10.1016/j.infsof.2026.108246,2026,Architectural perspectives on GenAI-driven software testing systems: A systematic literature review,Mariam Lahami,2,Information and Software Technology,0,
+W7162394858,10.48550/arxiv.2605.25536,10.48550/arxiv.2605.25536,2026,"A Tertiary Review of Large Language Model-Based Code Generating Tasks: Trends, Challenges, and Future Directions",Muslim Chochlov,3,arXiv (Cornell University),0,"Context. Large language models (LLMs) are increasingly applied to code-generating tasks (CGTs) in software engineering. While reported results are promising, the broader effects of such application and their integration into real-world development remain insufficiently understood with existing tertiary studies provide little in this area. Objective. This tertiary study consolidates secondary evidence on LLM-based CGTs, synthesizing the publication landscape, effects, scenarios, integration challenges, and future research directions. Method. Following systematic review guidelines, we searched in related digital libraries, complemented by backward-and-forward snowballing and screening step. Study quality was assessed and extraction reliability was audited with inter-rater agreement statistics. Evidence was synthesized using SWEBOK knowledge areas and the HELM framework. Results. We identify 30 secondary studies published between 2017-2025, with rapid growth since 2023. Accuracy seems strong on benchmarks but weakly supported for real-world generalization; robustness is fragile across tasks and configurations; efficiency constraints are pervasive; toxicity and bias are under-reported. Dominant challenges concern economic feasibility, evaluation validity, and socio-technical integration. Future directions suggest domain-aware model improvement and the need for holistic, standardized evaluation. Conclusion. LLM-based CGTs represent a fast-maturing yet unevenly evaluated research area, highlighting the need for domain-aware model improvements and holistic, standardized evaluation, addressing efficiency and associated costs."
+W7220650326,,10.5281/zenodo.23187741,2026,Bibliographic References Repository - Triaxial Evaluation of Large Language Models in Code Generation: A Structural and Semantic Approach.,Kleber Fernando Dos Santos,1,Zenodo (CERN European Organization for Nuclear Research),0,"This repository provides the bibliographic dataset and methodological documentation supporting the research article *""Triaxial Evaluation of Large Language Models in Code Generation: A Structural and Semantic Approach""*. The dataset consists of a comprehensive BibTeX (`.bib`) file containing a hybrid corpus of scientific literature focused on the evaluation of AI-generated code, specifically regarding structural complexity, semantic correctness, and the application of Large Language Models (LLMs). The bibliographic collection was constructed through a dual-methodology approach: 1. **Automated Search (IEEE Xplore):** Executed between April 6 and April 20, 2026, targeting metadata fields (Abstract and Document Title) for key benchmarks such as ""HumanEval"", ""SWE-bench"", ""LiveCodeBench"", as well as queries intersecting LLMs, evaluation metrics, and coding.2. **Manual Curation:** Integration of state-of-the-art preprints (ArXiv) concerning LLM-as-a-judge evaluation paradigms, alongside seminal software engineering literature detailing classical complexity metrics (e.g., McCabe's Cyclomatic Complexity, Halstead's Software Science). Essential technical documentation and official repositories for the evaluated tools are also included. The provided `.bib` file represents the raw, unfiltered extraction prior to the application of the research's strict exclusion criteria, which subsequently isolated papers exclusively addressing metrics for AI-generated code. A `README.md` file is included in this repository, detailing the exact search strings, temporal scope, and the exclusion methodology required to reproduce the literature review process."
+W7202261998,,10.5281/zenodo.21906011,2026,AI Agents in Support of Software Design and Architecture: Systematic Literature Review Protocol,RAMIRO DA CONCEIÇÃO DO NASCIMENTO JÚNIOR,11,Zenodo (CERN European Organization for Nuclear Research),0,"Protocol for a systematic literature review characterising the state ofthe art of LLM-based agents applied to software design and architecture,from two perspectives: the software engineering tasks addressed and theinternal architecture of the agents. The review follows Kitchenham andCharters (2007) with a hybrid search strategy and iterative snowballing(Wohlin, 2014), and is compatible with the PRISMA 2020 workflow. The search string was frozen on 10 July 2026 after achieving 100%sensitivity against a quasi-gold standard of 15 studies. This record is deposited before the screening phase begins. Subsequentamendments are published as new versions of this same record, preservingthe complete version history."
+W4416283130,10.48550/arxiv.2507.05270,10.48550/arxiv.2507.05270,2025,"Open Source, Hidden Costs: A Systematic Literature Review on OSS License Management","Li, Boyuan",6,arXiv (Cornell University),0,"Integrating third-party software components is a common practice in modern software development, offering significant advantages in terms of efficiency and innovation. However, this practice is fraught with risks related to software licensing. A lack of understanding may lead to disputes, which can pose serious legal and operational challenges. To these ends, both academia and industry have conducted various investigations and proposed solutions and tools to deal with these challenges. However, significant limitations still remain. Moreover, the rapid evolution of open-source software (OSS) licenses, as well as the rapidly incorporated generative software engineering techniques, such as large language models for code (CodeLLMs), are placing greater demands on the systematic management of software license risks. To unveil the severe challenges and explore possible future directions, we conduct the first systematic literature review (SLR) on 80 carefully selected OSS license-related papers, classifying existing research into three key categories, i.e., license identification, license risk assessment, and license risk mitigation. Based on these, we discuss challenges in existing solutions, conclude the opportunities to shed light on future research directions and offer practical recommendations for practitioners. We hope this thorough review will help bridge the gaps between academia and industry and accelerate the ecosystem-wide governance of legitimate software risks within the software engineering community."
+W7167467797,,10.59934/jaiea.v5i3.2431,2026,The Role of Generative AI in the Software Development Life Cycle: A Systematic Literature Review,Sebastian Saut Marulitua Sinaga,4,Journal of Artificial Intelligence and Engineering Applications (JAIEA),0,"The rapid advancement of Generative Artificial Intelligence (Generative AI), particularly through the emergence of Large Language Models (LLMs), has significantly transformed modern software engineering practices. These technologies enable automation across various phases of the Software Development Life Cycle (SDLC), including system design, coding, testing, and software maintenance. Despite their potential to improve development efficiency, the widespread adoption of Generative AI also introduces critical concerns related to software security, code quality, and long-term maintainability. This study aims to analyze the opportunities, security risks, and mitigation strategies associated with the integration of Generative AI into the SDLC. A Systematic Literature Review (SLR) with a qualitative descriptive approach was conducted by examining 15 primary studies published between 2021 and 2026, retrieved from IEEE Xplore, ACM Digital Library, Scopus, Google Scholar, and Portal Garuda. The collected literature was analyzed using content analysis and thematic analysis to identify the impacts of Generative AI across different SDLC phases. The findings reveal that Generative AI significantly enhances developer productivity, achieving efficiency gains of approximately 35% during system design, 55% during coding, 45% during testing, and 40% during software maintenance. However, AI-generated code remains vulnerable to various security weaknesses, including SQL Injection, Cross-Site Scripting (XSS), and improper input validation. Furthermore, excessive reliance on AI-generated outputs may contribute to technical debt accumulation through code duplication and reduced refactoring activities. To address these challenges, this study recommends the implementation of a Stratified AI-Human Governance Framework (SAHGF), which combines automated security validation, human code review, security testing, and continuous monitoring."
+W7125596406,,10.1145/3787202,2026,Survey on Learning-based Dynamic Fault Localization: From Traditional Machine Learning to Large Language Models,Chunyan Liu,6,ACM Computing Surveys,4,"Learning-based dynamic fault localization techniques play a crucial role in the field of software engineering. These techniques dynamically execute test cases to meticulously extract useful knowledge from the execution information in the program, with the aim of identifying fault locations by leveraging machine learning, deep learning, and large language models. Currently, there is already a flourishing body of research that is intensely focused on learning-based dynamic fault localization. Research literature can be categorized into two main aspects for learning-based dynamic fault localization: data-based enhancements (i.e., the datasets) and model-based enhancements (i.e., the suspiciousness algorithms). Thus, we conduct an extensive literature review on learning-based dynamic fault localization from the aspects of the data task and the model task. Among them, each task is divided into multiple sub-tasks in a systematic manner to comprehensively discuss the details. In addition, we analyze and summarize the datasets and metrics that have been widely used to evaluate the effectiveness of the proposed techniques in recent years, so that researchers can have an intuitive perception of them. We also discuss the present challenges and the directions for future research."
+W7133857781,,10.24406/publica-7754,2026,Quality assessment of software requirements using artificial intelligence methods: A systematic literature review,Elise Wolf,4,Fraunhofer-Publica (Fraunhofer-Gesellschaft),0,"Context: The quality of requirements specifications is a critical success factor in software development. Assuring high-quality requirements, specifically in an automated way, poses a significant challenge due to their unstructured and multi-modal character. With the rise of deep learning and large language models (LLMs), new opportunities have developed to assess the quality of requirements automatically, particularly user stories in the context of agile software engineering, where short development cycles require efficient tool support. Objective: This study aims to systematically review and investigate the current landscape of approaches based on artificial intelligence techniques such as natural language processing and deep learning for assessing the quality of software requirements. The investigation focuses on the artificial intelligence techniques adopted, quality aspects considered, datasets used to tune and evaluate the proposed approaches, and their performance. Method: We conducted a systematic literature review of 26 peer-reviewed papers published between 2019 and 2025. We selected the papers after a title and abstract review of 353 papers identified through a literature databases query and forward–backward snowballing. Results: The results reveal significant overlap among considered quality aspects, which can be mapped onto the higher-order requirements quality model INVEST. Most studies focus on assessing requirement quality rather than improving requirements and rely heavily on synthetic and public datasets. LLMs have rapidly gained popularity since 2023, though model evaluation strategies remain inconsistent. Metrics such as accuracy, precision, recall, and F1-Score are common, yet a few studies use semantic or expert-based evaluations. Conclusion: The field is evolving toward LLM-driven, semantically rich models, yet lacks methodological standardization, reproducible datasets for evaluating the models, and integration of the approaches with real-world requirements engineering processes. Future work should address these limitations by developing benchmark datasets, standardizing evaluation metrics, and exploring hybrid systems that combine AI-based and traditional requirements quality assurance approaches."
+W7213930371,,10.31673/2409-7292.2026.035907,2026,METHOD OF CONTEXT-DRIVEN DYNAMIC SOFTWARE SECURITY TESTING USING LARGE LANGUAGE MODEL-BASED AGENTS,M. V. Maksimovich,1,Modern information security,0,"The advancement of artificial intelligence methods, in particular large language models and the agent-basedsystems built upon them, opens new opportunities for improving automated software security testing. Among the existingmethods, dynamic application security testing (DAST) is the most promising direction for such improvement, since itperforms end-to-end verification of a real running application and reflects its actual exploitable behaviour. At the sametime, classical DAST tools construct attack scenarios through stochastic enumeration based on predefined wordlists,which results in incomplete coverage and considerable resource consumption. The aim of this work is to develop a methodof context-driven dynamic software security testing based on agents built upon large language models, which increasesthe completeness of vulnerability detection and reduces the resource cost of scanning. The proposed method uses theOpenAPI specification as a source of semantic context, on the basis of which agents generate a set of end-to-end (E2E)security tests that act as a deterministic source of traffic and steer the dynamic scan through real business processes insteadof random crawling. The method provides two improvements: extended coverage of vulnerabilities that are unreachablefor signature-based scanners, and a reduced volume of redundant requests, since only the generated traffic is scanned. Avulnerability coverage model, the workflow, and the test scenario generation strategy are described, and metrics for evaluating the effectiveness of the method are defined. The results indicate the feasibility of the agent-based approach forextending coverage and improving the resource efficiency of dynamic testing, and outline the direction of furtherexperimental research.Keywords: dynamic application security testing, large language models, agent-based systems, OpenAPI, BOLA,IDOR, test automation. References1. Qadir, S., Waheed, E., Khanum, A., & Jehan, S. (2025). Comparative evaluation of approaches & tools foreffective security testing of Web applications. PeerJ Computer Science, 11, e2821. https://doi.org/10.7717/peerj-cs.2821.2. Xu, H., Wang, S., Li, N., Wang, K., Zhao, Y., Chen, K., Yu, T., Liu, Y., & Wang, H. (2025). Large languagemodels for cyber security: A systematic literature review. ACM Transactions on Software Engineering and Methodology.https://doi.org/10.1145/3769676.3. Eriksson, B., Pellegrino, G., & Sabelfeld, A. (2021). Black Widow: Blackbox data-driven web scanning. In2021 IEEE Symposium on Security and Privacy (SP) (pp. 1125–1142). IEEE. https://doi.org/10.1109/ SP40001.2021.00022.4. Xue, J., He, Y., Xiao, Y., Chen, Y., Wang, Z., & Lu, H. (2025). From heuristics to intelligence: Evolving blackbox scanners for modern web applications. In 2025 IEEE 10th International Conference on Data Science in Cyberspace(DSC) (pp. 247–254). IEEE. https://doi.org/10.1109/DSC67331.2025.00039.5. OWASP Foundation. (2023). API1:2023 Broken object level authorization. OWASP API Security Top 10 –2023. https://owasp.org/API-Security/editions/2023/en/0xa1-broken-object-level-authorization/.6. Alazmi, S., & De Leon, D. C. (2022). A systematic literature review on the characteristics and effectiveness ofweb application vulnerability scanners. IEEE Access, 10, 33200–33219. https://doi.org/10.1109/ACCESS.2022.3161522.7. Millar, S., Podgurskii, D., Kuykendall, D., Martínez del Rincón, J., & Miller, P. (2022). Optimising vulnerabilitytriage in DAST with deep learning. In Proceedings of the 15th ACM Workshop on Artificial Intelligence and Security(pp. 137–147). ACM. https://doi.org/10.1145/3560830.3563724.8. Khare, A., Dutta, S., Li, Z., Solko-Breslin, A., Alur, R., & Naik, M. (2025). Understanding the effectiveness oflarge language models in detecting security vulnerabilities. In 2025 IEEE Conference on Software Testing, Verificationand Validation (ICST) (pp. 103–114). IEEE. https://doi.org/10.1109/ICST62969.2025.10988968.9. Gnieciak, D., & Szandała, T. (2025). Large language models versus static code analysis tools: A systematicbenchmark for vulnerability detection. IEEE Access, 13, 198410–198422. https://doi.org/10.1109/ ACCESS.2025.3635168.10. Happe, A., & Cito, J. (2023). Getting pwn’d by AI: Penetration testing with large language models. InProceedings of the 31st ACM Joint European Software Engineering Conference and Symposium on the Foundations ofSoftware Engineering (pp. 2082–2086). ACM. https://doi.org/10.1145/3611643.3613083.11. Deng, G., Liu, Y., Mayoral-Vilches, V., Liu, P., Li, Y., Xu, Y., Zhang, T., Liu, Y., Pinzger, M., & Rass, S.(2024). PentestGPT: Evaluating and harnessing large language models for automated penetration testing. In Proceedingsof the 33rd USENIX Security Symposium (pp. 847–864). USENIXLine truncated
+W7126280998,,,2026,Infusion of Blockchain to Establish Trustworthiness in AI Supported Software Evolution: A Systematic Literature Review,Mohammad Naserameri,2,arXiv (Cornell University),0,"Context: Blockchain and AI are increasingly explored to enhance trustworthiness in software engineering (SE), particularly in supporting software evolution tasks. Method: We conducted a systematic literature review (SLR) using a predefined protocol with clear eligibility criteria to ensure transparency, reproducibility, and minimized bias, synthesizing research on blockchain-enabled trust in AI-driven SE tools and processes. Results: Most studies focus on integrating AI in SE, with only 31% explicitly addressing trustworthiness. Our review highlights six recent studies exploring blockchain-based approaches to reinforce reliability, transparency, and accountability in AI-assisted SE tasks. Conclusion: Blockchain enhances trust by ensuring data immutability, model transparency, and lifecycle accountability, including federated learning with blockchain consensus and private data verification. However, inconsistent definitions of trust and limited real-world testing remain major challenges. Future work must develop measurable, reproducible trust frameworks to enable reliable, secure, and compliant AI-driven SE ecosystems, including applications involving large language models."
+W7126195749,10.48550/arxiv.2601.20918,10.48550/arxiv.2601.20918,2026,Infusion of Blockchain to Establish Trustworthiness in AI Supported Software Evolution: A Systematic Literature Review,Mohammad Naserameri,2,arXiv (Cornell University),0,"Context: Blockchain and AI are increasingly explored to enhance trustworthiness in software engineering (SE), particularly in supporting software evolution tasks. Method: We conducted a systematic literature review (SLR) using a predefined protocol with clear eligibility criteria to ensure transparency, reproducibility, and minimized bias, synthesizing research on blockchain-enabled trust in AI-driven SE tools and processes. Results: Most studies focus on integrating AI in SE, with only 31% explicitly addressing trustworthiness. Our review highlights six recent studies exploring blockchain-based approaches to reinforce reliability, transparency, and accountability in AI-assisted SE tasks. Conclusion: Blockchain enhances trust by ensuring data immutability, model transparency, and lifecycle accountability, including federated learning with blockchain consensus and private data verification. However, inconsistent definitions of trust and limited real-world testing remain major challenges. Future work must develop measurable, reproducible trust frameworks to enable reliable, secure, and compliant AI-driven SE ecosystems, including applications involving large language models."
+W7127972666,,10.70315/uloap.ulirs.2026.0301009,2026,Accelerating Mobile Application Development and Testing with Artificial Intelligence: A Systematic Literature Review,Maksym Yurko,1,Universal Library of Innovative Research and Studies,0,"The article presents a systematic literature review examining how artificial intelligence methods (LLM/GenAI, computer vision, deep learning, and multi-agent architectures) accelerate mobile application development and testing within the mobile SDLC. The objective is to address a deficit of domain-specific systematisation for mobile engineering and to answer three classes of questions: which AI approaches are applied across development and QA stages, which acceleration and efficiency metrics are empirically substantiated, and which quality/security risks accompany the adoption of generative tools. The relevance is driven by the growing complexity of mobile ecosystems and the limited scalability of manual testing and script-based automation, particularly due to brittleness under GUI changes. The novelty of the review lies in synthesising evidence from 28 selected studies from 2019 to 2025, with an explicit focus on mobile-specific constraints, and in juxtaposing speed gains against a trust contour. The principal findings are as follows: AI assistants demonstrate a substantial acceleration of routine tasks (up to ~55%) and a reduction in timelines for large-scale migrations. In testing, a transition toward LLM agents is observed, enabling high coverage at both scenario and element levels, as well as resilience to UI evolution. Concurrently, a trust crisis is documented due to a significant share of vulnerabilities in generated code, dependency hallucinations, and increased technical debt, necessitating the institutionalisation of verification practices and secure-by-default principles. The article is intended to be beneficial to software engineering researchers and to mobile development/QA practitioners integrating GenAI into workflows."
+W7213309362,,10.5281/zenodo.22771852,2026,From Vibe Coding to Cognitive Augmentation: Rethinking the Role of Large Language Models in Software Development and Developer Learning,Glorious Babalola,1,Zenodo (CERN European Organization for Nuclear Research),0,"Overview This publication presents Version 1.0 of an ongoing research project examining the changing relationship between large language models (LLMs), AI-assisted programming, software development, programming education, and human problem-solving. The rapid adoption of generative AI coding tools has introduced a significant shift in how software is learned and produced. Developers and students can increasingly delegate activities such as code generation, debugging, explanation, documentation, and problem decomposition to AI systems. While existing research has identified substantial benefits in areas such as programming productivity, learning support, and reduction of cognitive workload, growing research also raises questions about over-reliance, cognitive offloading, critical thinking, verification, and the development of independent programming ability. This paper begins from that tension. Rather than asking whether AI is simply ""good"" or ""bad"" for programmers, the research explores a more fundamental question: Under what conditions can AI-assisted programming improve productivity and learning while preserving and developing the independent problem-solving abilities of software developers and programming learners? Purpose of Version 1.0 This first version is not presented as a completed empirical study. Instead, it establishes the literature and conceptual foundation for a broader research program. The paper reviews and synthesizes existing literature concerning: Large language models and AI-assisted programming Generative AI in programming education Developer productivity and AI coding assistants Cognitive offloading and human–AI interaction Critical thinking and generative AI AI-assisted learning versus AI-assisted production The experiences and practices of developers using AI coding tools Risks associated with excessive dependence on AI-generated solutions Based on this literature, the paper introduces a preliminary conceptual framework for cognitive-preserving AI use in software development and developer learning. Proposed Research Direction The paper proposes that the central challenge is not whether developers should use AI, but how AI should be integrated into the development and learning process. The proposed approach distinguishes between AI use that substitutes for human reasoning and AI use that augments it. The preliminary framework, referred to as the Cognitive-Preserving AI Development Framework (CPADF), emphasizes the following cycle: Think → Attempt → Ask → Evaluate → Implement → Explain → Reproduce The framework is intended to encourage developers and learners to retain meaningful engagement with problem formulation, reasoning, debugging, evaluation, explanation, and independent reproduction, while still benefiting from the productivity and informational capabilities of AI systems. The framework is conceptual at this stage and should therefore be regarded as a research proposition requiring further investigation, rather than as a validated intervention or established methodology. Position Within the Broader Research Project This paper represents the first contribution to an evolving research project. Future stages of the project may investigate the proposed framework through additional literature review, expert evaluation, surveys, interviews, controlled experiments, observational studies, and other empirical methods. Potential areas for future investigation include: The relationship between AI-assisted programming and independent debugging ability. Differences between unrestricted AI use and structured, cognitive-preserving AI use. The effects of AI assistance on programming learning and long-term retention. Differences in AI use between novice, intermediate, and experienced developers. The relationship between confidence in AI-generated code and critical evaluation. The effects of repeated AI dependence on opportunities to practice programming problem-solving. Whether structured AI-use frameworks can preserve learning while maintaining productivity benefits. How these effects may differ across educational and professional contexts. Future versions may incorporate additional evidence, expert feedback, empirical findings, methodological refinements, and collaboration with researchers, educators, and software engineering practitioners. Intended Contribution The objective of this work is not to argue that AI-assisted programming is inherently harmful, nor to reject the use of AI coding tools. Instead, it seeks to contribute to an emerging discussion about human–AI collaboration in software development by proposing a distinction between cognitive substitution and cognitive augmentation. The broader goal is to explorLine truncated
+W7160995210,,10.1007/s10515-026-00629-6,2026,An exploratory study on effectiveness of GPT-4o in conducting sub-tasks of systematic literature reviews,Mohammad Shameem,6,Automated Software Engineering,2,
+W7165574043,,10.1016/j.infsof.2026.108232,2026,AI-driven self-healing across the edge–cloud continuum: A systematic literature review,Lander Bonilla,4,Information and Software Technology,2,"Context: The increasing complexity posed by Edge–Cloud Continuum architectures requires innovative software development practices and paradigms. Traditional software engineering practices fall short for addressing the particularities of such deployments; hence, there is a critical need for innovative tools and techniques capable of autonomous self-remediation and enhanced resilience with minimal human intervention. Objective: The overarching goal of this study is to synthesize the state-of-the-art with regard to AI-driven self-healing in the computing continuum. Implementation challenges and ethical implications are analyzed to identify effective approaches, architectures, and future trends in the field of automated program repair. Methods: A Systematic Literature Review was conducted following the PRISMA 2020 guidelines and Kitchenham’s methodology for software engineering. A rigorous automated search across designated academic databases initially yielded 2930 records. After applying strict inclusion and exclusion criteria, 99 primary studies were selected. Results: The analysis reveals a fundamental paradigm shift from observational monitoring to actionable, autonomous repair. While LLM-driven approaches and Agentic frameworks demonstrate high efficacy in tasks such as causal discovery and complex code patching, a critical validation gap remains regarding their deployment in non-deterministic runtime environments. Current literature often lacks the rigorous testing frameworks required to ensure that AI-generated repairs do not introduce regressions or security vulnerabilities. Conclusion: Achieving trustworthy autonomy requires a holistic approach that merges advanced AI paradigms with disciplined software engineering. Future research must prioritize the creation of standardized benchmarks and empirical studies to bridge the gap between theoretical algorithms and industrial application. Furthermore, the development of decentralized multi-agent architectures and Human-in-the-Loop techniques is essential to ensure safety, explainability, and ethical compliance in the next generation of resilient software systems."
+W7200674033,,,2025,Experiences with integrating AI tools into software engineering education: A literature review,Ilari Matikainen,1,Työväentutkimus Vuosikirja,0,"Due to landmark developments in large language model technologies during recent years, a large influx of novel artificial intelligence software has created newfound interest in utilizing AI tools for a variety of different use cases. This thesis looks into the early experiences of integrating AI tools into software engineering education, and examines the effects that their use has had on learning and teaching software engineering. The study is conducted through a literature review. The data used for the literature review is a collection of previous studies on the use of AI tools in the classroom, with focus on their practical applications. The main findings are that both ChatGPT and custom-developed educational AI tools are being utilized actively, and that their use has generally had a positive effect on multiple aspects of software engineering education. Negative effects were identified too, however. These findings suggest that artificial intelligence will most likely have a larger role in the future education of software engineers, and different ways of utilizing AI tools in educational interventions should be further researched."