Masters_Thesis/02-literature/search/results/L1-openalex-llm-for-lit-search.csv

source_id,arxiv,doi,year,title,first_author,n_authors,venue,citations,abstract
W4393119065,,10.1038/s41746-024-01157-x,2024,The ethics of ChatGPT in medicine and healthcare: a systematic review on Large Language Models (LLMs),Joschka Haltaufderheide,2,npj Digital Medicine,393,"With the introduction of ChatGPT, Large Language Models (LLMs) have received enormous attention in healthcare. Despite potential benefits, researchers have underscored various ethical implications. While individual instances have garnered attention, a systematic and comprehensive overview of practical applications currently researched and ethical issues connected to them is lacking. Against this background, this work maps the ethical landscape surrounding the current deployment of LLMs in medicine and healthcare through a systematic review. Electronic databases and preprint servers were queried using a comprehensive search strategy which generated 796 records. Studies were screened and extracted following a modified rapid review approach. Methodological quality was assessed using a hybrid approach. For 53 records, a meta-aggregative synthesis was performed. Four general fields of applications emerged showcasing a dynamic exploration phase. Advantages of using LLMs are attributed to their capacity in data analysis, information provisioning, support in decision-making or mitigating information loss and enhancing information accessibility. However, our study also identifies recurrent ethical concerns connected to fairness, bias, non-maleficence, transparency, and privacy. A distinctive concern is the tendency to produce harmful or convincing but inaccurate content. Calls for ethical guidance and human oversight are recurrent. We suggest that the ethical guidance debate should be reframed to focus on defining what constitutes acceptable human oversight across the spectrum of applications. This involves considering the diversity of settings, varying potentials for harm, and different acceptable thresholds for performance and certainty in healthcare. Additionally, critical inquiry is needed to evaluate the necessity and justification of LLMs' current experimental use."
W4406110856,,10.1073/pnas.2411962122,2025,Transforming literature screening: The emerging role of large language models in systematic reviews,Fernando M. Delgado-Chaves,8,Proceedings of the National Academy of Sciences,86,"Systematic reviews (SR) synthesize evidence-based medical literature, but they involve labor-intensive manual article screening. Large language models (LLMs) can select relevant literature, but their quality and efficacy are still being determined compared to humans. We evaluated the overlap between title- and abstract-based selected articles of 18 different LLMs and human-selected articles for three SR. In the three SRs, 185/4,662, 122/1,741, and 45/66 articles have been selected and considered for full-text screening by two independent reviewers. Due to technical variations and the inability of the LLMs to classify all records, the LLM's considered sample sizes were smaller. However, on average, the 18 LLMs classified 4,294 (min 4,130; max 4,329), 1,539 (min 1,449; max 1,574), and 27 (min 22; max 37) of the titles and abstracts correctly as either included or excluded for the three SRs, respectively. Additional analysis revealed that the definitions of the inclusion criteria and conceptual designs significantly influenced the LLM performances. In conclusion, LLMs can reduce one reviewer´s workload between 33% and 93% during title and abstract screening. However, the exact formulation of the inclusion and exclusion criteria should be refined beforehand for ideal support of the LLMs."
W4403583423,,10.2196/52597,2024,Large Language Models and Empathy: Systematic Review,Vera Sorin,7,Journal of Medical Internet Research,240,"BACKGROUND: Empathy, a fundamental aspect of human interaction, is characterized as the ability to experience another being's emotions within oneself. In health care, empathy is a fundamental for health care professionals and patients' interaction. It is a unique quality to humans that large language models (LLMs) are believed to lack. OBJECTIVE: We aimed to review the literature on the capacity of LLMs in demonstrating empathy. METHODS: We conducted a literature search on MEDLINE, Google Scholar, PsyArXiv, medRxiv, and arXiv between December 2022 and February 2024. We included English-language full-length publications that evaluated empathy in LLMs' outputs. We excluded papers evaluating other topics related to emotional intelligence that were not specifically empathy. The included studies' results, including the LLMs used, performance in empathy tasks, and limitations of the models, along with studies' metadata were summarized. RESULTS: A total of 12 studies published in 2023 met the inclusion criteria. ChatGPT-3.5 (OpenAI) was evaluated in all studies, with 6 studies comparing it with other LLMs such GPT-4, LLaMA (Meta), and fine-tuned chatbots. Seven studies focused on empathy within a medical context. The studies reported LLMs to exhibit elements of empathy, including emotions recognition and emotional support in diverse contexts. Evaluation metric included automatic metrics such as Recall-Oriented Understudy for Gisting Evaluation and Bilingual Evaluation Understudy, and human subjective evaluation. Some studies compared performance on empathy with humans, while others compared between different models. In some cases, LLMs were observed to outperform humans in empathy-related tasks. For example, ChatGPT-3.5 was evaluated for its responses to patients' questions from social media, where ChatGPT's responses were preferred over those of humans in 78.6% of cases. Other studies used subjective readers' assigned scores. One study reported a mean empathy score of 1.84-1.9 (scale 0-2) for their fine-tuned LLM, while a different study evaluating ChatGPT-based chatbots reported a mean human rating of 3.43 out of 4 for empathetic responses. Other evaluations were based on the level of the emotional awareness scale, which was reported to be higher for ChatGPT-3.5 than for humans. Another study evaluated ChatGPT and GPT-4 on soft-skills questions in the United States Medical Licensing Examination, where GPT-4 answered 90% of questions correctly. Limitations were noted, including repetitive use of empathic phrases, difficulty following initial instructions, overly lengthy responses, sensitivity to prompts, and overall subjective evaluation metrics influenced by the evaluator's background. CONCLUSIONS: LLMs exhibit elements of cognitive empathy, recognizing emotions and providing emotionally supportive responses in various contexts. Since social skills are an integral part of intelligence, these advancements bring LLMs closer to human-like interactions and expand their potential use in applications requiring emotional intelligence. However, there remains room for improvement in both the performance of these models and the evaluation strategies used for assessing soft skills."
W4385623654,,10.1101/2023.08.07.23293769,2023,Large Language Models (LLMs) and Empathy – A Systematic Review,Vera Sorin,7,medRxiv,45,"Abstract Purpose Empathy, a cornerstone of human interaction, is a unique quality to humans that Large Language Models (LLMs) are believed to lack. Our study aims to review the literature on the capacity of LLMs in demonstrating empathy Methods We conducted a literature search on MEDLINE up to July 2023. Seven publications ultimately met the inclusion criteria. Results All studies included in this review were published in 2023. All studies but one focused on ChatGPT-3.5 by OpenAI. Only one study evaluated empathy based on objective metrics, and all others used subjective human assessment. The studies reported LLMs to exhibits elements of empathy, including emotions recognition and providing emotionally supportive responses in diverse contexts, most of which were related to healthcare. In some cases, LLMs were observed to outperform humans in empathy-related tasks. Conclusion LLMs demonstrated some aspects of empathy in variable scenarios, mainly related to healthcare. The empathy may be considered “cognitive” empathy. Social skills are a fundamental aspect of intelligence, thus further research is imperative to enhance these skills in AI."
W4410157139,,10.1093/jamia/ocaf063,2025,The emergence of large language models as tools in literature reviews: a large language model-assisted systematic review,Dmitry Scherbakov,5,Journal of the American Medical Informatics Association,154,"OBJECTIVES: This study aims to summarize the usage of large language models (LLMs) in the process of creating a scientific review by looking at the methodological papers that describe the use of LLMs in review automation and the review papers that mention they were made with the support of LLMs. MATERIALS AND METHODS: The search was conducted in June 2024 in PubMed, Scopus, Dimensions, and Google Scholar by human reviewers. Screening and extraction process took place in Covidence with the help of LLM add-on based on the OpenAI GPT-4o model. ChatGPT and Scite.ai were used in cleaning the data, generating the code for figures, and drafting the manuscript. RESULTS: Of the 3788 articles retrieved, 172 studies were deemed eligible for the final review. ChatGPT and GPT-based LLM emerged as the most dominant architecture for review automation (n = 126, 73.2%). A significant number of review automation projects were found, but only a limited number of papers (n = 26, 15.1%) were actual reviews that acknowledged LLM usage. Most citations focused on the automation of a particular stage of review, such as Searching for publications (n = 60, 34.9%) and Data extraction (n = 54, 31.4%). When comparing the pooled performance of GPT-based and BERT-based models, the former was better in data extraction with a mean precision of 83.0% (SD = 10.4) and a recall of 86.0% (SD = 9.8). DISCUSSION AND CONCLUSION: Our LLM-assisted systematic review revealed a significant number of research projects related to review automation using LLMs. Despite limitations, such as lower accuracy of extraction for numeric data, we anticipate that LLMs will soon change the way scientific reviews are conducted."
W4402824743,,10.1093/jamiaopen/ooae098,2024,Literature search sandbox: a large language model that generates search queries for systematic reviews,Gaelen P. Adam,7,JAMIA Open,18,"Objectives: Development of search queries for systematic reviews (SRs) is time-consuming. In this work, we capitalize on recent advances in large language models (LLMs) and a relatively large dataset of natural language descriptions of reviews and corresponding Boolean searches to generate Boolean search queries from SR titles and key questions. Materials and Methods: We curated a training dataset of 10 346 SR search queries registered in PROSPERO. We used this dataset to fine-tune a set of models to generate search queries based on Mistral-Instruct-7b. We evaluated the models quantitatively using an evaluation dataset of 57 SRs and qualitatively through semi-structured interviews with 8 experienced medical librarians. Results: The model-generated search queries had median sensitivity of 85% (interquartile range [IQR] 40%-100%) and number needed to read of 1206 citations (IQR 205-5810). The interviews suggested that the models lack both the necessary sensitivity and precision to be used without scrutiny but could be useful for topic scoping or as initial queries to be refined. Discussion: Future research should focus on improving the dataset with more high-quality search queries, assessing whether fine-tuning the model on other fields, such as the population and intervention, improves performance, and exploring the addition of interactivity to the interface. Conclusions: The datasets developed for this project can be used to train and evaluate LLMs that map review descriptions to Boolean search queries. The models cannot replace thoughtful search query design but may be useful in providing suggestions for key words and the framework for the query."
W4394956892,,10.1111/medu.15402,2024,A systematic review of large language models and their implications in medical education,Harrison Lucas,3,Medical Education,284,"INTRODUCTION: In the past year, the use of large language models (LLMs) has generated significant interest and excitement because of their potential to revolutionise various fields, including medical education for aspiring physicians. Although medical students undergo a demanding educational process to become competent health care professionals, the emergence of LLMs presents a promising solution to challenges like information overload, time constraints and pressure on clinical educators. However, integrating LLMs into medical education raises critical concerns and challenges for educators, professionals and students. This systematic review aims to explore LLM applications in medical education, specifically their impact on medical students' learning experiences. METHODS: A systematic search was performed in PubMed, Web of Science and Embase for articles discussing the applications of LLMs in medical education using selected keywords related to LLMs and medical education, from the time of ChatGPT's debut until February 2024. Only articles available in full text or English were reviewed. The credibility of each study was critically appraised by two independent reviewers. RESULTS: The systematic review identified 166 studies, of which 40 were found by review to be relevant to the study. Among the 40 relevant studies, key themes included LLM capabilities, benefits such as personalised learning and challenges regarding content accuracy. Importantly, 42.5% of these studies specifically evaluated LLMs in a novel way, including ChatGPT, in contexts such as medical exams and clinical/biomedical information, highlighting their potential in replicating human-level performance in medical knowledge. The remaining studies broadly discussed the prospective role of LLMs in medical education, reflecting a keen interest in their future potential despite current constraints. CONCLUSIONS: The responsible implementation of LLMs in medical education offers a promising opportunity to enhance learning experiences. However, ensuring information accuracy, emphasising skill-building and maintaining ethical safeguards are crucial. Continuous critical evaluation and interdisciplinary collaboration are essential for the appropriate integration of LLMs in medical education."
W4401671212,,10.1016/j.csi.2024.103917,2024,Large language models for code completion: A systematic literature review,Rasha Ahmad Husein,3,Computer Standards & Interfaces,72,"Code completion serves as a fundamental aspect of modern software development, improving developers' coding processes. Integrating code completion tools into an Integrated Development Environment (IDE) or code editor enhances the coding process and boosts productivity by reducing errors and speeding up code writing while reducing cognitive load. This is achieved by predicting subsequent tokens, such as keywords, variable names, types, function names, operators, and more. Different techniques can achieve code completion, and recent research has focused on Deep Learning methods, particularly Large Language Models (LLMs) utilizing Transformer algorithms. While several research papers have focused on the use of LLMs for code completion, these studies are fragmented, and there is no systematic overview of the use of LLMs for code completion. Therefore, we aimed to perform a Systematic Literature Review (SLR) study to investigate how LLMs have been applied for code completion so far. We have formulated several research questions to address how LLMs have been integrated for code completion-related tasks and to assess the efficacy of these LLMs in the context of code completion. To achieve this, we retrieved 244 papers from scientific databases using auto-search and specific keywords, finally selecting 23 primary studies based on an SLR methodology for in-depth analysis. This SLR study categorizes the granularity levels of code completion achieved by utilizing LLMs in IDEs, explores the existing issues in current code completion systems, how LLMs address these challenges, and the pre-training and fine-tuning methods employed. Additionally, this study identifies open research problems and outlines future research directions. Our analysis reveals that LLMs significantly enhance code completion performance across several programming languages and contexts, and their capability to predict relevant code snippets based on context and partial input boosts developer productivity substantially."
W4402231520,,10.2139/ssrn.4943759,2024,Accuracy of Large Language Models for Literature Screening in Systematic Reviews and Meta-Analyses,Zhang-Yi Dai,6,SSRN Electronic Journal,6,
W4402193006,,10.2196/57400,2024,Large Language Models for Mental Health Applications: Systematic Review,Zhijun Guo,6,JMIR Mental Health,269,"BACKGROUND: Large language models (LLMs) are advanced artificial neural networks trained on extensive datasets to accurately understand and generate natural language. While they have received much attention and demonstrated potential in digital health, their application in mental health, particularly in clinical settings, has generated considerable debate. OBJECTIVE: This systematic review aims to critically assess the use of LLMs in mental health, specifically focusing on their applicability and efficacy in early screening, digital interventions, and clinical settings. By systematically collating and assessing the evidence from current studies, our work analyzes models, methodologies, data sources, and outcomes, thereby highlighting the potential of LLMs in mental health, the challenges they present, and the prospects for their clinical use. METHODS: Adhering to the PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) guidelines, this review searched 5 open-access databases: MEDLINE (accessed by PubMed), IEEE Xplore, Scopus, JMIR, and ACM Digital Library. Keywords used were (mental health OR mental illness OR mental disorder OR psychiatry) AND (large language models). This study included articles published between January 1, 2017, and April 30, 2024, and excluded articles published in languages other than English. RESULTS: In total, 40 articles were evaluated, including 15 (38%) articles on mental health conditions and suicidal ideation detection through text analysis, 7 (18%) on the use of LLMs as mental health conversational agents, and 18 (45%) on other applications and evaluations of LLMs in mental health. LLMs show good effectiveness in detecting mental health issues and providing accessible, destigmatized eHealth services. However, assessments also indicate that the current risks associated with clinical use might surpass their benefits. These risks include inconsistencies in generated text; the production of hallucinations; and the absence of a comprehensive, benchmarked ethical framework. CONCLUSIONS: This systematic review examines the clinical applications of LLMs in mental health, highlighting their potential and inherent risks. The study identifies several issues: the lack of multilingual datasets annotated by experts, concerns regarding the accuracy and reliability of generated content, challenges in interpretability due to the ""black box"" nature of LLMs, and ongoing ethical dilemmas. These ethical concerns include the absence of a clear, benchmarked ethical framework; data privacy issues; and the potential for overreliance on LLMs by both physicians and patients, which could compromise traditional medical practices. As a result, LLMs should not be considered substitutes for professional mental health services. However, the rapid development of LLMs underscores their potential as valuable clinical aids, emphasizing the need for continued research and development in this area. TRIAL REGISTRATION: PROSPERO CRD42024508617; https://www.crd.york.ac.uk/prospero/display_record.php?RecordID=508617."
W4403087472,,10.2196/22769,2024,Applications and Concerns of ChatGPT and Other Conversational Large Language Models in Health Care: Systematic Review,Leyao Wang,8,Journal of Medical Internet Research,172,"BACKGROUND: The launch of ChatGPT (OpenAI) in November 2022 attracted public attention and academic interest to large language models (LLMs), facilitating the emergence of many other innovative LLMs. These LLMs have been applied in various fields, including health care. Numerous studies have since been conducted regarding how to use state-of-the-art LLMs in health-related scenarios. OBJECTIVE: This review aims to summarize applications of and concerns regarding conversational LLMs in health care and provide an agenda for future research in this field. METHODS: We used PubMed, ACM, and the IEEE digital libraries as primary sources for this review. We followed the guidance of PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) to screen and select peer-reviewed research articles that (1) were related to health care applications and conversational LLMs and (2) were published before September 1, 2023, the date when we started paper collection. We investigated these papers and classified them according to their applications and concerns. RESULTS: Our search initially identified 820 papers according to targeted keywords, out of which 65 (7.9%) papers met our criteria and were included in the review. The most popular conversational LLM was ChatGPT (60/65, 92% of papers), followed by Bard (Google LLC; 1/65, 2% of papers), LLaMA (Meta; 1/65, 2% of papers), and other LLMs (6/65, 9% papers). These papers were classified into four categories of applications: (1) summarization, (2) medical knowledge inquiry, (3) prediction (eg, diagnosis, treatment recommendation, and drug synergy), and (4) administration (eg, documentation and information collection), and four categories of concerns: (1) reliability (eg, training data quality, accuracy, interpretability, and consistency in responses), (2) bias, (3) privacy, and (4) public acceptability. There were 49 (75%) papers using LLMs for either summarization or medical knowledge inquiry, or both, and there are 58 (89%) papers expressing concerns about either reliability or bias, or both. We found that conversational LLMs exhibited promising results in summarization and providing general medical knowledge to patients with a relatively high accuracy. However, conversational LLMs such as ChatGPT are not always able to provide reliable answers to complex health-related tasks (eg, diagnosis) that require specialized domain expertise. While bias or privacy issues are often noted as concerns, no experiments in our reviewed papers thoughtfully examined how conversational LLMs lead to these issues in health care research. CONCLUSIONS: Future studies should focus on improving the reliability of LLM applications in complex health-related tasks, as well as investigating the mechanisms of how LLM applications bring bias and privacy issues. Considering the vast accessibility of LLMs, legal, social, and technical efforts are all needed to address concerns about LLMs to promote, improve, and regularize the application of LLMs in health care."
W4399977125,,10.3389/fpsyt.2024.1422807,2024,Applications of large language models in psychiatry: a systematic review,Mahmud Omar,6,Frontiers in Psychiatry,85,"Background: With their unmatched ability to interpret and engage with human language and context, large language models (LLMs) hint at the potential to bridge AI and human cognitive processes. This review explores the current application of LLMs, such as ChatGPT, in the field of psychiatry. Methods: We followed PRISMA guidelines and searched through PubMed, Embase, Web of Science, and Scopus, up until March 2024. Results: From 771 retrieved articles, we included 16 that directly examine LLMs' use in psychiatry. LLMs, particularly ChatGPT and GPT-4, showed diverse applications in clinical reasoning, social media, and education within psychiatry. They can assist in diagnosing mental health issues, managing depression, evaluating suicide risk, and supporting education in the field. However, our review also points out their limitations, such as difficulties with complex cases and potential underestimation of suicide risks. Conclusion: Early research in psychiatry reveals LLMs' versatile applications, from diagnostic support to educational roles. Given the rapid pace of advancement, future investigations are poised to explore the extent to which these models might redefine traditional roles in mental health care."
W4395688696,,10.1101/2024.04.26.24306390,2024,A Systematic Review of ChatGPT and Other Conversational Large Language Models in Healthcare,Leyao Wang,8,medRxiv,41,"Background: The launch of the Chat Generative Pre-trained Transformer (ChatGPT) in November 2022 has attracted public attention and academic interest to large language models (LLMs), facilitating the emergence of many other innovative LLMs. These LLMs have been applied in various fields, including healthcare. Numerous studies have since been conducted regarding how to employ state-of-the-art LLMs in health-related scenarios to assist patients, doctors, and public health administrators. Objective: This review aims to summarize the applications and concerns of applying conversational LLMs in healthcare and provide an agenda for future research on LLMs in healthcare. Methods: , 2023, the date when we started paper collection and screening. We investigated these papers and classified them according to their applications and concerns. Results: Our search initially identified 820 papers according to targeted keywords, out of which 65 papers met our criteria and were included in the review. The most popular conversational LLM was ChatGPT from OpenAI (60), followed by Bard from Google (1), Large Language Model Meta AI (LLaMA) from Meta (1), and other LLMs (5). These papers were classified into four categories in terms of their applications: 1) summarization, 2) medical knowledge inquiry, 3) prediction, and 4) administration, and four categories of concerns: 1) reliability, 2) bias, 3) privacy, and 4) public acceptability. There are 49 (75%) research papers using LLMs for summarization and/or medical knowledge inquiry, and 58 (89%) research papers expressing concerns about reliability and/or bias. We found that conversational LLMs exhibit promising results in summarization and providing medical knowledge to patients with a relatively high accuracy. However, conversational LLMs like ChatGPT are not able to provide reliable answers to complex health-related tasks that require specialized domain expertise. Additionally, no experiments in our reviewed papers have been conducted to thoughtfully examine how conversational LLMs lead to bias or privacy issues in healthcare research. Conclusions: Future studies should focus on improving the reliability of LLM applications in complex health-related tasks, as well as investigating the mechanisms of how LLM applications brought bias and privacy issues. Considering the vast accessibility of LLMs, legal, social, and technical efforts are all needed to address concerns about LLMs to promote, improve, and regularize the application of LLMs in healthcare."
W4393318738,,10.1186/s12909-024-05239-y,2024,Large language models for generating medical examinations: systematic review,Yaara Artsi,6,BMC Medical Education,126,"BACKGROUND: Writing multiple choice questions (MCQs) for the purpose of medical exams is challenging. It requires extensive medical knowledge, time and effort from medical educators. This systematic review focuses on the application of large language models (LLMs) in generating medical MCQs. METHODS: The authors searched for studies published up to November 2023. Search terms focused on LLMs generated MCQs for medical examinations. Non-English, out of year range and studies not focusing on AI generated multiple-choice questions were excluded. MEDLINE was used as a search database. Risk of bias was evaluated using a tailored QUADAS-2 tool. RESULTS: Overall, eight studies published between April 2023 and October 2023 were included. Six studies used Chat-GPT 3.5, while two employed GPT 4. Five studies showed that LLMs can produce competent questions valid for medical exams. Three studies used LLMs to write medical questions but did not evaluate the validity of the questions. One study conducted a comparative analysis of different models. One other study compared LLM-generated questions with those written by humans. All studies presented faulty questions that were deemed inappropriate for medical exams. Some questions required additional modifications in order to qualify. CONCLUSIONS: LLMs can be used to write MCQs for medical examinations. However, their limitations cannot be ignored. Further study in this field is essential and more conclusive evidence is needed. Until then, LLMs may serve as a supplementary tool for writing medical examinations. 2 studies were at high risk of bias. The study followed the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) guidelines."
W4414761564,,10.1136/rapm-2025-107204,2025,Beware of large language models for scoping literature searches,Ryan David Williams,3,Regional Anesthesia & Pain Medicine,0,"We read with interest the article by Dr Purwel and colleagues on the potential applications of large language models (LLMs) in meta-analyses and systematic reviews.[1][1] While we agree that LLMs could significantly reduce the time required for the laborious components of systematic reviews, we wish"
W4406658975,,10.1038/s43856-024-00717-2,2025,Current applications and challenges in large language models for patient care: a systematic review,Felix Busch,14,Communications Medicine,355,"BACKGROUND: The introduction of large language models (LLMs) into clinical practice promises to improve patient education and empowerment, thereby personalizing medical care and broadening access to medical knowledge. Despite the popularity of LLMs, there is a significant gap in systematized information on their use in patient care. Therefore, this systematic review aims to synthesize current applications and limitations of LLMs in patient care. METHODS: We systematically searched 5 databases for qualitative, quantitative, and mixed methods articles on LLMs in patient care published between 2022 and 2023. From 4349 initial records, 89 studies across 29 medical specialties were included. Quality assessment was performed using the Mixed Methods Appraisal Tool 2018. A data-driven convergent synthesis approach was applied for thematic syntheses of LLM applications and limitations using free line-by-line coding in Dedoose. RESULTS: We show that most studies investigate Generative Pre-trained Transformers (GPT)-3.5 (53.2%, n = 66 of 124 different LLMs examined) and GPT-4 (26.6%, n = 33/124) in answering medical questions, followed by patient information generation, including medical text summarization or translation, and clinical documentation. Our analysis delineates two primary domains of LLM limitations: design and output. Design limitations include 6 second-order and 12 third-order codes, such as lack of medical domain optimization, data transparency, and accessibility issues, while output limitations include 9 second-order and 32 third-order codes, for example, non-reproducibility, non-comprehensiveness, incorrectness, unsafety, and bias. CONCLUSIONS: This review systematically maps LLM applications and limitations in patient care, providing a foundational framework and taxonomy for their implementation and evaluation in healthcare settings."
W4413394768,,10.1109/access.2025.3601206,2025,A Systematic Literature Review of Hallucinations in Large Language Models,Christian Woesle,3,IEEE Access,22,"This review systematically maps research on hallucinations in large language models using a descriptive scheme that links model outputs to four system architectures: unaugmented generation, post-hoc reactive validation, proactive detection-and-mitigation, and fully integrated detection-and-mitigation designs. Our methodology for this systematic review follows the PRISMA guidelines to ensure transparency and reproducibility. We searched IEEE Xplore, ACM Digital Library, and ScienceDirect for studies published between 2015 and January 2025 and extracted 125 peer-reviewed papers across nine application domains. Quantitative analysis shows that question answering and multimodal tasks account for 48% of all papers, whereas software engineering, educational technology, and autonomous systems are underexplored. Although 87.5% of the studies rely on additional reactive or proactive defenses, only 8.8% implement integrated architecture-level safeguards, revealing a critical gap in unified and dynamic architectures. The resulting classification matrix and domain map provide a diagnostic tool for locating blind spots and comparing architectural maturity. Three actionable priorities emerge: develop integrated reasoning-and-verification loops that pre-empt hallucinations; transfer proven causal-intervention and multi-agent validation pipelines to high-stakes, under-represented domains and benchmark them under real conditions; and build modular, cross-domain evaluation frameworks that isolate the contribution of individual mitigation components and support ablation studies. By consolidating fragmented evidence and quantifying architecture-domain imbalances, this review establishes a traceable foundation for engineering reliable, explainable, and domain-adaptable countermeasures to hallucinations in generative language technology."
W4400414272,,10.1001/jamanetworkopen.2024.20496,2024,Performance of a Large Language Model in Screening Citations,Takehiko Oami,3,JAMA Network Open,85,"Importance: Large language models (LLMs) are promising as tools for citation screening in systematic reviews. However, their applicability has not yet been determined. Objective: To evaluate the accuracy and efficiency of an LLM in title and abstract literature screening. Design, Setting, and Participants: This prospective diagnostic study used the data from the title and abstract screening process for 5 clinical questions (CQs) in the development of the Japanese Clinical Practice Guidelines for Management of Sepsis and Septic Shock. The LLM decided to include or exclude citations based on the inclusion and exclusion criteria in terms of patient, population, problem; intervention; comparison; and study design of the selected CQ and was compared with the conventional method for title and abstract screening. This study was conducted from January 7 to 15, 2024. Exposures: LLM (GPT-4 Turbo)-assisted citation screening or the conventional method. Main Outcomes and Measures: The sensitivity and specificity of the LLM-assisted screening process was calculated, and the full-text screening result using the conventional method was set as the reference standard in the primary analysis. Pooled sensitivity and specificity were also estimated, and screening times of the 2 methods were compared. Results: In the conventional citation screening process, 8 of 5634 publications in CQ 1, 4 of 3418 in CQ 2, 4 of 1038 in CQ 3, 17 of 4326 in CQ 4, and 8 of 2253 in CQ 5 were selected. In the primary analysis of 5 CQs, LLM-assisted citation screening demonstrated an integrated sensitivity of 0.75 (95% CI, 0.43 to 0.92) and specificity of 0.99 (95% CI, 0.99 to 0.99). Post hoc modifications to the command prompt improved the integrated sensitivity to 0.91 (95% CI, 0.77 to 0.97) without substantially compromising specificity (0.98 [95% CI, 0.96 to 0.99]). Additionally, LLM-assisted screening was associated with reduced time for processing 100 studies (1.3 minutes vs 17.2 minutes for conventional screening methods; mean difference, -15.25 minutes [95% CI, -17.70 to -12.79 minutes]). Conclusions and Relevance: In this prospective diagnostic study investigating the performance of LLM-assisted citation screening, the model demonstrated acceptable sensitivity and reasonably high specificity with reduced processing time. This novel method could potentially enhance efficiency and reduce workload in systematic reviews."
W4398223047,,10.3390/jcm13113041,2024,Clinical and Surgical Applications of Large Language Models: A Systematic Review,Sophia M. Pressman,6,Journal of Clinical Medicine,106,"Background: Large language models (LLMs) represent a recent advancement in artificial intelligence with medical applications across various healthcare domains. The objective of this review is to highlight how LLMs can be utilized by clinicians and surgeons in their everyday practice. Methods: A systematic review was conducted following the Preferred Reporting Items for Systematic Reviews and Meta-Analyses guidelines. Six databases were searched to identify relevant articles. Eligibility criteria emphasized articles focused primarily on clinical and surgical applications of LLMs. Results: The literature search yielded 333 results, with 34 meeting eligibility criteria. All articles were from 2023. There were 14 original research articles, four letters, one interview, and 15 review articles. These articles covered a wide variety of medical specialties, including various surgical subspecialties. Conclusions: LLMs have the potential to enhance healthcare delivery. In clinical settings, LLMs can assist in diagnosis, treatment guidance, patient triage, physician knowledge augmentation, and administrative tasks. In surgical settings, LLMs can assist surgeons with documentation, surgical planning, and intraoperative guidance. However, addressing their limitations and concerns, particularly those related to accuracy and biases, is crucial. LLMs should be viewed as tools to complement, not replace, the expertise of healthcare professionals."
W4407873728,,10.7326/annals-24-02189,2025,Development of Prompt Templates for Large Language Model–Driven Screening in Systematic Reviews,Christian Cao,19,Annals of Internal Medicine,72,"BACKGROUND: Systematic reviews (SRs) are hindered by the initial rigorous article screen, which delays access to reliable information synthesis. OBJECTIVE: To develop generic prompt templates for large language model (LLM)-driven abstract and full-text screening that can be adapted to different reviews. DESIGN: Diagnostic test accuracy. SETTING: 48 425 citations were tested for abstract screening across 10 SRs. Full-text screening evaluated all 12 690 freely available articles from the original search. Prompt development used the GPT4-0125-preview model (OpenAI). PARTICIPANTS: None. MEASUREMENTS: Large language models were prompted to include or exclude articles based on SR eligibility criteria. Model outputs were compared with original SR author decisions after full-text screening to evaluate performance (accuracy, sensitivity, and specificity). RESULTS: Optimized prompts using GPT4-0125-preview achieved a weighted sensitivity of 97.7% (range, 86.7% to 100%) and specificity of 85.2% (range, 68.3% to 95.9%) in abstract screening and weighted sensitivity of 96.5% (range, 89.7% to 100.0%) and specificity of 91.2% (range, 80.7% to 100%) in full-text screening across 10 SRs. In contrast, zero-shot prompts had poor sensitivity (49.0% abstract, 49.1% full-text). Across LLMs, Claude-3.5 (Anthropic) and GPT4 variants had similar performance, whereas Gemini Pro (Google) and GPT3.5 (OpenAI) models underperformed. Direct screening costs for 10 000 citations differed substantially: Where single human abstract screening was estimated to require more than 83 hours and $1666.67 USD, our LLM-based approach completed screening in under 1 day for $157.02 USD. LIMITATIONS: Further prompt optimizations may exist. Retrospective study. Convenience sample of SRs. Full-text screening evaluations were limited to free PubMed Central full-text articles. CONCLUSION: A generic prompt for abstract and full-text screening achieving high sensitivity and specificity that can be adapted to other SRs and LLMs was developed. Our prompting innovations may have value to SR investigators and researchers conducting similar criteria-based tasks across the medical sciences. PRIMARY FUNDING SOURCE: None."
W7128199163,,10.5220/0013562900003967,2025,Evaluating Large Language Models for Literature Screening: A Systematic Review of Sensitivity and Workload Reduction,Elias Sandner,8,"International Conference on Data Science, Technology and Applications (DATA)",4,
W4407764671,,10.2196/67488,2025,Accuracy of Large Language Models for Literature Screening in Thoracic Surgery: Diagnostic Study,Zhang-Yi Dai,7,Journal of Medical Internet Research,29,"BACKGROUND: Systematic reviews and meta-analyses rely on labor-intensive literature screening. While machine learning offers potential automation, its accuracy remains suboptimal. This raises the question of whether emerging large language models (LLMs) can provide a more accurate and efficient approach. OBJECTIVE: This paper evaluates the sensitivity, specificity, and summary receiver operating characteristic (SROC) curve of LLM-assisted literature screening. METHODS: We conducted a diagnostic study comparing the accuracy of LLM-assisted screening versus manual literature screening across 6 thoracic surgery meta-analyses. Manual screening by 2 investigators served as the reference standard. LLM-assisted screening was performed using ChatGPT-4o (OpenAI) and Claude-3.5 (Anthropic) sonnet, with discrepancies resolved by Gemini-1.5 pro (Google). In addition, 2 open-source, machine learning-based screening tools, ASReview (Utrecht University) and Abstrackr (Center for Evidence Synthesis in Health, Brown University School of Public Health), were also evaluated. We calculated sensitivity, specificity, and 95% CIs for the title and abstract, as well as full-text screening, generating pooled estimates and SROC curves. LLM prompts were revised based on a post hoc error analysis. RESULTS: LLM-assisted full-text screening demonstrated high pooled sensitivity (0.87, 95% CI 0.77-0.99) and specificity (0.96, 95% CI 0.91-0.98), with the area under the curve (AUC) of 0.96 (95% CI 0.94-0.97). Title and abstract screening achieved a pooled sensitivity of 0.73 (95% CI 0.57-0.85) and specificity of 0.99 (95% CI 0.97-0.99), with an AUC of 0.97 (95% CI 0.96-0.99). Post hoc revisions improved sensitivity to 0.98 (95% CI 0.74-1.00) while maintaining high specificity (0.98, 95% CI 0.94-0.99). In comparison, the pooled sensitivity and specificity of ASReview tool-assisted screening were 0.58 (95% CI 0.53-0.64) and 0.97 (95% CI 0.91-0.99), respectively, with an AUC of 0.66 (95% CI 0.62-0.70). The pooled sensitivity and specificity of Abstrackr tool-assisted screening were 0.48 (95% CI 0.35-0.62) and 0.96 (95% CI 0.88-0.99), respectively, with an AUC of 0.78 (95% CI 0.74-0.82). A post hoc meta-analysis revealed comparable effect sizes between LLM-assisted and conventional screening. CONCLUSIONS: LLMs hold significant potential for streamlining literature screening in systematic reviews, reducing workload without sacrificing quality. Importantly, LLMs outperformed traditional machine learning-based tools (ASReview and Abstrackr) in both sensitivity and AUC values, suggesting that LLMs offer a more accurate and efficient approach to literature screening."
W7211887844,,10.17802/2306-1278-2026-15-4-153-163,2026,USING LARGE LANGUAGE MODELS FOR LITERATURE SEARCH IN CARDIOVASCULAR SURGERY SYSTEMATIC REVIEWS AND META-ANALYSES,A. S. Shatskiy,4,Complex Issues of Cardiovascular Diseases,0,"Highlights Large Language Models (LLMs) enable automation of initial abstract screening in systematic reviews, significantly reducing manual workload. The effectiveness of LLMs heavily depends on prompt engineering, which must clearly translate inclusion and exclusion criteria into actionable instructions. Two prompt design strategies are presented and illustrated: a “lenient” approach (maximizing recall) and a “strict” approach (reducing false positives). A practical workflow for downloading and processing PubMed abstracts using LLMs (e.g., DeepSeek, GPT‑4o, Perplexity) is demonstrated. Key limitations of LLMs – hallucinations, sensitivity to phrasing, and training‑data biases — are discussed, emphasizing the need for expert validation. A two‑stage screening strategy is recommended: first a lenient pass, followed by a stricter filter, with subsequent manual verification of relevant articles. Prompt engineering is framed as an iterative, domain‑specific art rather than a one‑size‑fits‑all procedure, requiring continuous refinement for each research question. Abstract Large Language Models have evolved into powerful tools for automating the primary screening of articles’ abstracts in systematic reviews, enabling a significant reduction in manual labor. The article presents a comprehensive review of prompt engineering principles, demonstrating how traditional meta-analysis criteria can be transformed into clear instructions for artificial intelligence. Using case studies in coronary artery bypass grafting and congenital heart disease surgery, we illustrate the impact of prompt formulation on the comprehensiveness, accuracy, and overall efficiency of literature screening. Furthermore, typical errors are discussed, and the ongoing necessity of expert oversight to minimize hallucinations and biases inherent in artificial intelligence conclusions is emphasized. Ultimately, systematic prompt engineering combined with expert evaluation allows researchers in cardiovascular surgery to optimize the search for meta-analysis sources based on Large Language Models, ensuring a faster and more comprehensive synthesis of evidence and methods for processing primary material."
W4410169548,,10.3390/biomedinformatics5020025,2025,Performance Comparison of Large Language Models for Efficient Literature Screening,Maria Teresa Colangelo,5,BioMedInformatics,6,"Background: Systematic reviewers face a growing body of biomedical literature, making early-stage article screening increasingly time-consuming. In this study, we assessed six large language models (LLMs)—OpenHermes, Flan T5, GPT-2, Claude 3 Haiku, GPT-3.5 Turbo, and GPT-4o—for their ability to identify randomized controlled trials (RCTs) in datasets of increasing difficulty. Methods: We first retrieved articles from PubMed and used all-mpnet-base-v2 to measure semantic similarity to known target RCTs, stratifying the collection into quartiles of descending relevance. Each LLM then received either verbose or concise prompts to classify articles as “Accepted” or “Rejected”. Results: Claude 3 Haiku, GPT-3.5 Turbo, and GPT-4o consistently achieved high recall, though their precision varied in the quartile with the highest similarity, where false positives increased. By contrast, smaller or older models struggled to balance sensitivity and specificity, with some over-including irrelevant studies or missing key articles. Importantly, multi-stage prompts did not guarantee performance gains for weaker models, whereas single-prompt approaches proved effective for advanced LLMs. Conclusions: These findings underscore that both model capability and prompt design strongly affect classification outcomes, suggesting that newer LLMs, if properly guided, can substantially expedite systematic reviews."
W4405623173,,10.2196/66648,2024,Large Language Models in Gastroenterology: Systematic Review,Eun Jeong Gong,8,Journal of Medical Internet Research,37,"BACKGROUND: As health care continues to evolve with technological advancements, the integration of artificial intelligence into clinical practices has shown promising potential to enhance patient care and operational efficiency. Among the forefront of these innovations are large language models (LLMs), a subset of artificial intelligence designed to understand, generate, and interact with human language at an unprecedented scale. OBJECTIVE: This systematic review describes the role of LLMs in improving diagnostic accuracy, automating documentation, and advancing specialist education and patient engagement within the field of gastroenterology and gastrointestinal endoscopy. METHODS: Core databases including MEDLINE through PubMed, Embase, and Cochrane Central registry were searched using keywords related to LLMs (from inception to April 2024). Studies were included if they satisfied the following criteria: (1) any type of studies that investigated the potential role of LLMs in the field of gastrointestinal endoscopy or gastroenterology, (2) studies published in English, and (3) studies in full-text format. The exclusion criteria were as follows: (1) studies that did not report the potential role of LLMs in the field of gastrointestinal endoscopy or gastroenterology, (2) case reports and review papers, (3) ineligible research objects (eg, animals or basic research), and (4) insufficient data regarding the potential role of LLMs. Risk of Bias in Non-Randomized Studies-of Interventions was used to evaluate the quality of the identified studies. RESULTS: Overall, 21 studies on the potential role of LLMs in gastrointestinal disorders were included in the systematic review, and narrative synthesis was done because of heterogeneity in the specified aims and methodology in each included study. The overall risk of bias was low in 5 studies and moderate in 16 studies. The ability of LLMs to spread general medical information, offer advice for consultations, generate procedure reports automatically, or draw conclusions about the presumptive diagnosis of complex medical illnesses was demonstrated by the systematic review. Despite promising benefits, such as increased efficiency and improved patient outcomes, challenges related to data privacy, accuracy, and interdisciplinary collaboration remain. CONCLUSIONS: We highlight the importance of navigating these challenges to fully leverage LLMs in transforming gastrointestinal endoscopy practices. TRIAL REGISTRATION: PROSPERO 581772; https://www.crd.york.ac.uk/prospero/."
W7138936899,,10.47989/ir31iconf64265,2026,Collaborating with large language models in literature screening for a systematic review of college students’ GenAI literacy,Wonchan Choi,5,Information Research,0,"Introduction. Literature screening is among the most time- and labour-intensive phases of systematic literature reviews (SLRs). Although large language models (LLMs) have been explored for screening, prior work has focused on medical and environmental domains and mainly benchmarked LLMs against human coders, offering limited guidance on collaborative integration in SLRs. Method. This study evaluated 12 GPT model–prompt configurations in an SLR of 1,616 publications. Two human coders screened a 10% sample (n = 162) to create a gold standard for model comparison. Performance was assessed using balanced accuracy, recall for inclusion, time efficiency, and cost efficiency. Disagreements between humans and models were analysed. Analysis. Bootstrap tests compared performance across configurations. Open coding identified error types in human–model disagreements. Group discussion resolved discrepancies. Results. GPT-5 Zeroshot and GPT-4o-mini Fewshot achieved the highest performance (accuracy = 0.990 and 0.946; recall = 1.000 and 0.933). GPT-4o-mini was faster and cheaper but more prone to overly rigid rule application. Error analysis identified 10 mismatches, leading to two corrections of human miscoding. Conclusion. LLM-assisted screening can reduce workload, improve efficiency, and correct human errors in SLR. Practical guidelines for prompt design and confidence thresholds can position LLMs as collaborative tools in SLRs."
W4404402066,,10.2196/59607,2024,Examining the Role of Large Language Models in Orthopedics: Systematic Review,Cheng Zhang,8,Journal of Medical Internet Research,64,"BACKGROUND: Large language models (LLMs) can understand natural language and generate corresponding text, images, and even videos based on prompts, which holds great potential in medical scenarios. Orthopedics is a significant branch of medicine, and orthopedic diseases contribute to a significant socioeconomic burden, which could be alleviated by the application of LLMs. Several pioneers in orthopedics have conducted research on LLMs across various subspecialties to explore their performance in addressing different issues. However, there are currently few reviews and summaries of these studies, and a systematic summary of existing research is absent. OBJECTIVE: The objective of this review was to comprehensively summarize research findings on the application of LLMs in the field of orthopedics and explore the potential opportunities and challenges. METHODS: PubMed, Embase, and Cochrane Library databases were searched from January 1, 2014, to February 22, 2024, with the language limited to English. The terms, which included variants of ""large language model,"" ""generative artificial intelligence,"" ""ChatGPT,"" and ""orthopaedics,"" were divided into 2 categories: large language model and orthopedics. After completing the search, the study selection process was conducted according to the inclusion and exclusion criteria. The quality of the included studies was assessed using the revised Cochrane risk-of-bias tool for randomized trials and CONSORT-AI (Consolidated Standards of Reporting Trials-Artificial Intelligence) guidance. Data extraction and synthesis were conducted after the quality assessment. RESULTS: A total of 68 studies were selected. The application of LLMs in orthopedics involved the fields of clinical practice, education, research, and management. Of these 68 studies, 47 (69%) focused on clinical practice, 12 (18%) addressed orthopedic education, 8 (12%) were related to scientific research, and 1 (1%) pertained to the field of management. Of the 68 studies, only 8 (12%) recruited patients, and only 1 (1%) was a high-quality randomized controlled trial. ChatGPT was the most commonly mentioned LLM tool. There was considerable heterogeneity in the definition, measurement, and evaluation of the LLMs' performance across the different studies. For diagnostic tasks alone, the accuracy ranged from 55% to 93%. When performing disease classification tasks, ChatGPT with GPT-4's accuracy ranged from 2% to 100%. With regard to answering questions in orthopedic examinations, the scores ranged from 45% to 73.6% due to differences in models and test selections. CONCLUSIONS: LLMs cannot replace orthopedic professionals in the short term. However, using LLMs as copilots could be a potential approach to effectively enhance work efficiency at present. More high-quality clinical trials are needed in the future, aiming to identify optimal applications of LLMs and advance orthopedics toward higher efficiency and precision."
W4394890393,,10.1101/2024.04.15.24305869,2024,A Systematic Review of Testing and Evaluation of Healthcare Applications of Large Language Models (LLMs),Suhana Bedi,19,medRxiv,48,"1 Abstract Importance Large Language Models (LLMs) can assist in a wide range of healthcare-related activities. Current approaches to evaluating LLMs make it difficult to identify the most impactful LLM application areas. Objective To summarize the current evaluation of LLMs in healthcare in terms of 5 components: evaluation data type, healthcare task, Natural Language Processing (NLP)/Natural Language Understanding (NLU) task, dimension of evaluation, and medical specialty. Data Sources A systematic search of PubMed and Web of Science was performed for studies published between 01-01-2022 and 02-19-2024. Study Selection Studies evaluating one or more LLMs in healthcare. Data Extraction and Synthesis Three independent reviewers categorized 519 studies in terms of data used in the evaluation, the healthcare tasks (the what) and the NLP/NLU tasks (the how) examined, the dimension(s) of evaluation, and the medical specialty studied. Results Only 5% of reviewed studies utilized real patient care data for LLM evaluation. The most popular healthcare tasks were assessing medical knowledge (e.g. answering medical licensing exam questions, 44.5%), followed by making diagnoses (19.5%), and educating patients (17.7%). Administrative tasks such as assigning provider billing codes (0.2%), writing prescriptions (0.2%), generating clinical referrals (0.6%) and clinical notetaking (0.8%) were less studied. For NLP/NLU tasks, the vast majority of studies examined question answering (84.2%). Other tasks such as summarization (8.9%), conversational dialogue (3.3%), and translation (3.1%) were infrequent. Almost all studies (95.4%) used accuracy as the primary dimension of evaluation; fairness, bias and toxicity (15.8%), robustness (14.8%), deployment considerations (4.6%), and calibration and uncertainty (1.2%) were infrequently measured. Finally, in terms of medical specialty area, most studies were in internal medicine (42%), surgery (11.4%) and ophthalmology (6.9%), with nuclear medicine (0.6%), physical medicine (0.4%) and medical genetics (0.2%) being the least represented. Conclusions and Relevance Existing evaluations of LLMs mostly focused on accuracy of question answering for medical exams, without consideration of real patient care data. Dimensions like fairness, bias and toxicity, robustness, and deployment considerations received limited attention. To draw meaningful conclusions and improve LLM adoption, future studies need to establish a standardized set of LLM applications and evaluation dimensions, perform evaluations using data from routine care, and broaden testing to include administrative tasks as well as multiple medical specialties. Key Points Question How are healthcare applications of large language models (LLMs) currently evaluated? Findings Studies rarely used real patient care data for LLM evaluation. Administrative tasks such as generating provider billing codes and writing prescriptions were understudied. Natural Language Processing (NLP)/Natural Language Understanding (NLU) tasks like summarization, conversational dialogue, and translation were infrequently explored. Accuracy was the predominant dimension of evaluation, while fairness, bias and toxicity assessments were neglected. Evaluations in specialized fields, such as nuclear medicine and medical genetics were rare. Meaning Current LLM assessments in healthcare remain shallow and fragmented. To draw concrete insights on their performance, evaluations need to use real patient care data across a broad range of healthcare and NLP/NLU tasks and medical specialties with standardized dimensions of evaluation."
W4411259709,,10.1101/2025.06.13.25329541,2025,Automation of Systematic Reviews with Large Language Models,Christian Cao,36,medRxiv,33,"Abstract Importance Systematic reviews (SRs) inform evidence-based decision making. Yet, many take over a year to complete, are labor intensive, prone to human error, and face reproducibility challenges; thus limiting access to timely and reliable information. Objective To validate a large language model (LLM)-based workflow (otto-SR) to automate three of the most labour intensive tasks in performing SR’s: article screening, data extraction, and risk of bias assessment; and to assess its feasibility in rapidly updating existing reviews. Design, setting, and participants We conducted a validation study in four phases, with direct benchmarking against graduate-level human researchers in phases 1 and 2. Phase 1: article screening performance was measured across 32,357 citations from 5 systematic reviews. The reference standard consisted of the original reviews’ screening decisions after full-text screening. Phase 2: data extraction performance was measured across 4,495 data points from 495 studies in 7 reviews. Phase 3: risk of bias assessment (ROB2, Newcastle-Ottawa, QUADAS2) performance was measured across 345 studies from 12 reviews. Reference standards for Phase 2 and Phase 3 were created after blinded adjudication of the original review extraction and RoB assessments. Phase 4: otto-SR was used to reproduce and update the primary analysis from an issue of Cochrane reviews (n=12 reviews, 146,276 citations), with analytical comparisons to the original meta-analyzed findings. All discrepancies underwent dual human review. Results otto-SR showed high performance in phase 1 article screening ( otto-SR : 96.7% sensitivity, 97.9% specificity; human: 81.7% sensitivity, 98.1% specificity) and phase 2 data extraction ( otto-SR : 93.1% accuracy; human: 79.7% accuracy). In phase 3, otto-SR demonstrated high interrater reliability for risk of bias judgements (ROB2 0.98, Newcastle-Ottawa 0.95, QUADAS2 0.74; Gwet AC2). In phase 4, otto-SR , reproduced and updated the primary analysis from an issue of Cochrane reviews. Across Cochrane reviews, otto-SR incorrectly excluded a median of 0 studies (IQR 0 to 0.25), and found nearly twice as many eligible studies compared to the original authors (n= 114 vs. 64). Meta-analyses based on otto-SR generated screening and extraction outputs, subsequently verified through dual human review, yielded newly statistically significant effect estimates in 2 reviews and negated significance in 1 review. Conclusions and relevance LLMs have high performance in article screening, data extraction, and risk of bias assessments. They can rapidly reproduce and update existing systematic reviews, laying the foundation for automated, scalable, and reliable evidence synthesis."
W7133133003,,10.17605/osf.io/56d8j,2025,Performance of Large Language Models in Automated Medical Literature Screening: A Systematic Review and Meta-analysis,Chenggong Xie,1,,0,"This project aims to systematically evaluate the performance, reliability, and heterogeneity of large language models (LLMs) in the automated screening of medical literature. As the volume of scientific publications continues to grow exponentially, the screening phase of systematic reviews has become increasingly time-consuming and labor-intensive. Recent advances in artificial intelligence—particularly large language models such as GPT-series, LLaMA, Claude, Gemini, Copilot, DeepSeek, and Qwen—have shown the potential to automate key steps of evidence synthesis. However, the accuracy, consistency, and generalizability of these models across different screening tasks and clinical domains remain unclear. This study will conduct a comprehensive systematic review and meta-analysis to summarize existing evidence on the use of LLMs for title/abstract screening, citation filtering, and study selection in medical and health-related systematic reviews. We will assess multiple performance indicators, including sensitivity, specificity, precision, recall, F1-score, workload reduction, and time efficiency. We also aim to explore sources of heterogeneity, such as model type, prompt design, clinical field, screening task complexity, dataset characteristics, and evaluation methodology. The review will follow PRISMA 2020 guidelines. Searches will be conducted across PubMed, Embase, Scopus, Web of Science, and preprint servers. Eligible studies will include evaluations of LLM-assisted or fully automated literature screening workflows, using either real-world systematic review datasets or curated benchmark sets. Risk of bias will be assessed using adapted tools for AI performance studies. Where feasible, meta-analysis will be performed to synthesize quantitative performance metrics."
W4392791588,,10.1002/jrsm.1715,2024,Can large language models replace humans in systematic reviews? Evaluating GPT ‐4's efficacy in screening and extracting data from peer‐reviewed and grey literature in multiple languages,Qusai Khraisha,5,Research Synthesis Methods,266,"Systematic reviews are vital for guiding practice, research and policy, although they are often slow and labour-intensive. Large language models (LLMs) could speed up and automate systematic reviews, but their performance in such tasks has yet to be comprehensively evaluated against humans, and no study has tested Generative Pre-Trained Transformer (GPT)-4, the biggest LLM so far. This pre-registered study uses a ""human-out-of-the-loop"" approach to evaluate GPT-4's capability in title/abstract screening, full-text review and data extraction across various literature types and languages. Although GPT-4 had accuracy on par with human performance in some tasks, results were skewed by chance agreement and dataset imbalance. Adjusting for these caused performance scores to drop across all stages: for data extraction, performance was moderate, and for screening, it ranged from none in highly balanced literature datasets (~1:1) to moderate in those datasets where the ratio of inclusion to exclusion in studies was imbalanced (~1:3). When screening full-text literature using highly reliable prompts, GPT-4's performance was more robust, reaching ""human-like"" levels. Although our findings indicate that, currently, substantial caution should be exercised if LLMs are being used to conduct systematic reviews, they also offer preliminary evidence that, for certain review tasks delivered under specific conditions, LLMs can rival human performance."
W7138880922,,10.64898/2026.03.17.26348656,2026,Performance of Large Language Models in Automated Medical Literature Screening: A Systematic Review and Meta-analysis,Xie Chenggong,7,medRxiv,0,"Abstract Objective To systematically evaluate the diagnostic performance of large language models (LLMs) in automated medical literature screening and to determine their potential role in supporting evidence synthesis workflows. Methods A systematic review and meta-analysis was conducted according to PRISMA DTA guidance. PubMed, Web of Science, Embase, the Cochrane Library and Google Scholar were searched from 1 January 2022 to 17 November 2025. Studies assessing LLMs for automated title and abstract screening or full-text eligibility assessment in medical literature were included. Diagnostic accuracy metrics were extracted and pooled using a bivariate random effects model and hierarchical summary receiver operating characteristic (HSROC) analysis. Subgroup analyses and meta-regression were performed to explore sources of heterogeneity. Results Eighteen studies published between 2023 and 2025 were included. In title and abstract screening, the pooled sensitivity was 0.92 and pooled specificity was 0.94. The SROC area under the curve (AUC) reached 0.98. In full-text screening, pooled sensitivity and specificity both reached 0.99 and the AUC was 0.99. Prompt strategies incorporating examples or chain-of-thought reasoning significantly improved sensitivity. Across studies, most models were deployed without task specific fine tuning and still achieved strong performance. Subgroup analyses and meta regression did not identify significant sources of heterogeneity. Many studies also reported substantial efficiency gains, including large reductions in screening workload, time and cost. Conclusion LLMs demonstrate high diagnostic accuracy for automated medical literature screening, particularly in full-text assessment. These models show strong potential as high sensitivity assistive tools that can substantially reduce manual screening burden while supporting evidence synthesis. Further methodological optimization and validation in large scale real-world settings are required to establish their long term role in evidence-based medicine."
W7171023253,,10.1111/jebm.70166,2026,Performance of Large Language Models in Automated Medical Literature Screening: A Systematic Review and Meta‐Analysis,Chenggong Xie,7,Journal of Evidence-Based Medicine,0,"OBJECTIVE: To systematically evaluate the diagnostic performance of large language models (LLMs) in automated medical literature screening and to determine their potential role in supporting evidence synthesis workflows. METHODS: PubMed, Web of Science, Embase, The Cochrane Library, Google Scholar, CNKI, Wanfang, VIP, and CBM were searched from January 1, 2022 to June 11, 2026. Studies assessing LLMs for automated title and abstract screening or full-text eligibility assessment in medical literature were included. The primary outcomes were sensitivity and specificity. Secondary outcomes included positive and negative likelihood ratios, diagnostic odds ratio, area under the curve (AUC), and efficiency-related metrics. Pooled sensitivity and specificity were estimated using a bivariate random-effects model and hierarchical summary receiver operating characteristic framework. Subgroup analyses and meta-regression were performed to explore sources of heterogeneity. This systematic review was registered with the Open Science Framework (https://osf.io/56d8j). RESULTS: Eighteen studies published between 2023 and 2025 were included. In title and abstract screening, the pooled sensitivity was 0.92 (95% confidence interval [CI]: 0.81-0.96) and pooled specificity was 0.94 (95% CI: 0.90-0.97). The summary receiver operating characteristic AUC reached 0.98 (95% CI: 0.96-0.99). In full-text screening, pooled sensitivity and specificity both reached 0.99 (95% CI: 0.95-1.00) and the AUC was 0.99 (95% CI: 0.98-1.00). Prompt strategies incorporating examples or chain-of-thought reasoning were associated with higher sensitivity than strategies without these approaches (0.95 vs. 0.86, p < 0.01). Several studies reported substantial efficiency gains, including workload reductions ranging from approximately 50% to 99% and screening time reductions of up to tenfold. CONCLUSIONS: LLMs shows promising performance in automated medical literature screening, particularly in full-text assessment. These models show strong potential as high sensitivity assistive tools that can substantially reduce manual screening burden while supporting evidence synthesis. Further real-world validation is needed to establish their role in evidence-based medicine."
W4390127073,,10.1016/j.jval.2023.09.030,2023,P21 A Comparative Analysis of Large Language Models (LLM) Utilised in Systematic Literature Review,H Rathi,4,Value in Health,4,
W4410000915,,10.3390/info16050378,2025,"Large Language Models in Systematic Review Screening: Opportunities, Challenges, and Methodological Considerations",Carlo Galli,3,Information,35,"Large Language Models (LLMs) have recently emerged as a powerful option for partially automating the labor-intensive process of screening articles in systematic reviews. Unlike traditional semi-automated platforms that rely on iterative human feedback, LLM-based pipelines can operate in a zero-shot or few-shot manner, classifying abstracts according to predefined criteria. This paper offers a step-by-step methodology for researchers, librarians, and students seeking to incorporate LLMs—such as GPT-4—into systematic reviews. It discusses required software and data preprocessing, presents various prompt strategies, and emphasizes the importance of human oversight to maintain rigorous quality control. The proposed framework aims to provide best practices and offer guidance on managing costs, reproducibility, and prompt refinement. By following these guidelines, review teams can substantially reduce screening workloads without compromising the comprehensive nature of evidence-based research."
W4400099044,,10.3389/fdsfr.2024.1379260,2024,Assessing the performance of large language models in literature screening for pharmacovigilance: a comparative study,Dan Li,8,Frontiers in Drug Safety and Regulation,18,"Pharmacovigilance plays a crucial role in ensuring the safety of pharmaceutical products. It involves the systematic monitoring of adverse events and the detection of potential safety concerns related to drugs. Manual literature screening for pharmacovigilance related articles is a labor-intensive and time-consuming task, requiring streamlined solutions to cope with the continuous growth of literature. The primary objective of this study is to assess the performance of Large Language Models (LLMs) in automating literature screening for pharmacovigilance, aiming to enhance the process by identifying relevant articles more effectively. This study represents a novel application of LLMs including OpenAI's GPT-3.5, GPT-4, and Anthropic's Claude2, in the field of pharmacovigilance, evaluating their ability to categorize medical publications as relevant or irrelevant for safety signal reviews. Our analysis encompassed N-shot learning, chain-of-thought reasoning, and evaluating metrics, with a focus on factors impacting accuracy. The findings highlight the promising potential of LLMs in literature screening, achieving a reproducibility of 93%, sensitivity of 97%, and specificity of 67% showcasing notable strengths in terms of reproducibility and sensitivity, although with moderate specificity. Notably, performance improved when models were provided examples consisting of abstracts, labels, and corresponding reasoning explanations. Moreover, our exploration identified several potential contributing factors influencing prediction outcomes. These factors encompassed the choice of key words and prompts, the balance of the examples, and variations in reasoning explanations. By configuring advanced LLMs for efficient screening of extensive literature databases, this study underscores the transformative potential of these models in drug safety monitoring. Furthermore, these insights gained from this study can inform the development of automated systems for pharmacovigilance, contributing to the ongoing efforts to ensure the safety and efficacy of pharmacovigilance products."
W4411040734,,10.1016/j.jdent.2025.105877,2025,Large language models for the screening step in systematic reviews in dentistry,Rata Rokhshad,5,Journal of Dentistry,8,
W4395052272,,10.1016/j.isci.2024.109713,2024,The application of large language models in medicine: A scoping review,Xiangbin Meng,25,iScience,414,"This study systematically reviewed the application of large language models (LLMs) in medicine, analyzing 550 selected studies from a vast literature search. LLMs like ChatGPT transformed healthcare by enhancing diagnostics, medical writing, education, and project management. They assisted in drafting medical documents, creating training simulations, and streamlining research processes. Despite their growing utility in assisted diagnosis and improving doctor-patient communication, challenges persisted, including limitations in contextual understanding and the risk of over-reliance. The surge in LLM-related research indicated a focus on medical writing, diagnostics, and patient communication, but highlighted the need for careful integration, considering validation, ethical concerns, and the balance with traditional medical practice. Future research directions suggested a focus on multimodal LLMs, deeper algorithmic understanding, and ensuring responsible, effective use in healthcare."
W4415594129,,10.2196/76947,2025,Evaluating Large Language Models in Ophthalmology: Systematic Review,Zili Zhang,7,Journal of Medical Internet Research,34,"BACKGROUND Large language models (LLMs) have the potential to revolutionize ophthalmic care, but their evaluation practice remains fragmented. A systematic assessment is crucial to identify gaps and guide future evaluation practices and clinical integration. OBJECTIVE This study aims to map the current landscape of LLM evaluations in ophthalmology and explore whether performance synthesis is feasible for a common task. METHODS A comprehensive search of PubMed, Web of Science, Embase, and IEEE Xplore was conducted up to November 17, 2024 (no language limits). Eligible publications quantitatively assessed an existing or modified LLM on ophthalmology-related tasks. Studies without full-text availability or those focusing solely on vision-only models were excluded. Two reviewers screened studies and extracted data across 6 dimensions (evaluated LLM, data modality, ophthalmic subspecialty, medical task, evaluation dimension, and clinical alignment), and disagreements were resolved by a third reviewer. Descriptive statistics were analyzed and visualized using Python (with NumPy, Pandas, SciPy, and Matplotlib libraries). The Fisher exact test compared open- versus closed-source models. An exploratory random-effects meta-analysis (logit transformation; DerSimonian-Laird τ2) was performed for the diagnosis-making task; heterogeneity was reported with I2 and subgrouped by model, modality, and subspecialty. RESULTS Of the 817 identified records, 187 studies met the inclusion criteria. Closed-source LLMs dominated: 170 for ChatGPT, 58 for Gemini, and 32 for Copilot. Open-source LLMs appeared in only 25 (13.4%) of studies overall, but they appeared in 17 (77.3%) of evaluation-after-development studies, versus 8 (4.8%) pure-evaluation studies (P<1×10-5). Evaluations were chiefly text-only (n=168); image-text tasks, despite the centrality of imaging, were used in 19 studies. Subspecialty coverage was skewed toward comprehensive ophthalmology (n=72), retina and vitreous (n=32), and glaucoma (n=20). Refractive surgery, ocular pathology and oncology, and ophthalmic pharmacology each appeared in 3 or fewer studies. Medical query (n=86), standardized examination (n=41), and diagnosis making (n=29) emerged as the 3 predominant tasks, while research assistance (n=5), patient triaging (n=3), and disease prediction (n=3) received less attention. Accuracy was reported in most studies (n=176), whereas calibration and uncertainty were almost absent (n=5). Real-world patient data (n=45), human performance comparison (n=63), non‑English testing (n=24), and real-world deployment (n=4) were relatively absent. Exploratory meta-analysis pooled 28 diagnostic evaluations from 17 studies: overall accuracy was 0.594 (95% CI 0.488-0.692) with extreme heterogeneity (I2=94.5%). Subgroups remained heterogeneous (I2>80%), and findings were inconsistent (eg, pooled GPT-3.5>GPT-4). CONCLUSIONS Evidence on LLM evaluations in ophthalmology is extensive but heterogeneous. Most studies have tested a few closed-source LLMs on text-based questions, leaving open-source systems, multimodal tasks, non-English contexts, and real-world deployment underexamined. High methodological variability precludes meaningful performance aggregation, as illustrated by the heterogeneous meta-analysis. Standardized, multimodal benchmarks and phased clinical validation pipelines are urgently needed before LLMs can be safely integrated into eye care workflows."
W7129376919,,10.64898/2026.02.07.26345640,2026,Collaborative large language models (LLMs) are all you need for screening in systematic reviews,Mihir Parmar,16,medRxiv,1,"Background: The ability of large language models (LLMs) to work collaboratively and screen studies in a systematic review (SR) is under-explored. Hence, we aimed to evaluate the effectiveness of LLMs in automating the process of screening in systematic reviews. Methods: This is an observational study which included labeled data (title and abstracts) for five SRs. Originally, two reviewers screened the citations independently for eligibility. A third reviewer cross-checked each citation for quality assurance. GPT-4, Claude-3-Sonnet, and Gemini-Pro-1.0 were used using zero-shot chain-of-thought prompting. Collaborative approaches included (i): conflict resolution using benefit of the doubt, (ii) majority voting using an independent third LLM and (iii) conflict resolution using an informed third LLM. Performance was assessed using accuracy, precision for exclusion, and recall for inclusion. Work saved over samples (WSS) was computed to estimate the reduction in manual human effort. Results: A total of 11300 articles were included in this study. The individual models, GPT-4, Claude-3-Sonnet, and Gemini-Pro-1.0 exhibited a high precision for exclusion, achieving 99.7%, 99.7%, and 99.2% and high recall for inclusion achieving 95.5%, 96.6% and 85.7%, respectively. However, the collaborative approach utilizing the two best-performing models (GPT-4 and Claude-3S) achieved an average precision of 99.9% and a recall of 98.5% (across all collaborative approaches). Furthermore, the proposed collaborative approach resulted in an average WSS of 63.5%, compared to the average WSS of 45.2% for individual models. Conversational LLM interactions showed a consistent pattern of results. Limitations: This study was limited due to reliance on proprietary models, and evaluation on oncology datasets. Conclusion: Evidence shows that collaborative LLMs enable efficient, high-performing screening in systematic reviews, supporting continuous evidence updates. Primary funding source: NIH (U24CA265879-01-1) and Carolyn-Ann-Kennedy-Bacon Fund."
W4408837107,,10.1016/j.bjoms.2025.03.006,2025,The impact of the large language model ChatGPT in oral and maxillofacial surgery: a systematic review,Lucas Mariz de Menezes Torres,5,British Journal of Oral and Maxillofacial Surgery,12,
W7213443824,,10.1108/ajim-06-2025-0363,2026,Large language models for literature screening in conceptually complex and interdisciplinary reviews,Ni Cheng,5,Aslib Journal of Information Management,0,"Purpose This study examines the capabilities and limitations of large language models (LLMs) for literature screening in systematic reviews involving conceptually diffuse and interdisciplinary topics. Design/methodology/approach Two screening datasets were constructed. Four LLMs, Claude 4.6, DeepSeek V4 Pro, Gemini 3.1 Pro, and GPT-5.5, were evaluated using a staged, rule-guided prompting workflow covering title screening, abstract screening, and full-text assessment. Model performance was assessed using accuracy, precision, recall, specificity, the F1-score, and Cohen's kappa, supplemented by stage-wise screening flow analysis and qualitative discrepancy analysis between model and human reviewer decisions. Findings Model performance varied substantially across datasets and models. In the conceptually diffuse Dataset A, all models achieved high specificity, but positive-class performance was more limited. DeepSeek V4 Pro achieved the highest accuracy, precision, F1-score, and Cohen's kappa, whereas Claude 4.6 achieved the highest recall. In the more clearly bounded Dataset B, recall was higher and model behavior was more convergent, with GPT-5.5 showing the best overall balance across performance indicators. Stage-wise analysis showed that models differed in where they made inclusion and exclusion decisions, while discrepancy analysis indicated that errors were mainly related to conceptual scope confusion, context misidentification, and underestimation of analytical depth. These findings suggest that current LLMs can support literature screening, but their reliability depends strongly on the conceptual clarity of the review task and the structure of eligibility criteria. Originality/value This study extends the literature on LLM-assisted screening by focusing on conceptually abstract and interdisciplinary review tasks. This study introduces a staged, Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA)-aligned screening workflow and conducts an error analysis that explains how LLM screening can fail in semantically diffuse settings. The study provides practical implications for the transparent and responsible use of LLMs in informetric research and systematic review practice."
W4407873703,,10.7326/annals-25-00012,2025,Large Language Models for Screening Search Results in Systematic Reviews: Are We There Yet?,S. Swaroop Vedula,2,Annals of Internal Medicine,1,
W4399210824,,10.2196/56780,2024,Potential Roles of Large Language Models in the Production of Systematic Reviews and Meta-Analyses,Xufei Luo,10,Journal of Medical Internet Research,76,"Large language models (LLMs) such as ChatGPT have become widely applied in the field of medical research. In the process of conducting systematic reviews, similar tools can be used to expedite various steps, including defining clinical questions, performing the literature search, document screening, information extraction, and language refinement, thereby conserving resources and enhancing efficiency. However, when using LLMs, attention should be paid to transparent reporting, distinguishing between genuine and false content, and avoiding academic misconduct. In this viewpoint, we highlight the potential roles of LLMs in the creation of systematic reviews and meta-analyses, elucidating their advantages, limitations, and future research directions, aiming to provide insights and guidance for authors planning systematic reviews and meta-analyses."
W4401759841,,10.1186/s13643-024-02609-x,2024,Evaluating the effectiveness of large language models in abstract screening: a comparative analysis,Michael Li,3,Systematic Reviews,127,"OBJECTIVE: This study aimed to evaluate the performance of large language models (LLMs) in the task of abstract screening in systematic review and meta-analysis studies, exploring their effectiveness, efficiency, and potential integration into existing human expert-based workflows. METHODS: We developed automation scripts in Python to interact with the APIs of several LLM tools, including ChatGPT v4.0, ChatGPT v3.5, Google PaLM 2, and Meta Llama 2, and latest tools including ChatGPT v4.0 turbo, ChatGPT v3.5 turbo, Google Gemini 1.0 pro, Meta Llama 3, and Claude 3. This study focused on three databases of abstracts and used them as benchmarks to evaluate the performance of these LLM tools in terms of sensitivity, specificity, and overall accuracy. The results of the LLM tools were compared to human-curated inclusion decisions, gold standard for systematic review and meta-analysis studies. RESULTS: Different LLM tools had varying abilities in abstract screening. Chat GPT v4.0 demonstrated remarkable performance, with balanced sensitivity and specificity, and overall accuracy consistently reaching or exceeding 90%, indicating a high potential for LLMs in abstract screening tasks. The study found that LLMs could provide reliable results with minimal human effort and thus serve as a cost-effective and efficient alternative to traditional abstract screening methods. CONCLUSION: While LLM tools are not yet ready to completely replace human experts in abstract screening, they show great promise in revolutionizing the process. They can serve as autonomous AI reviewers, contribute to collaborative workflows with human experts, and integrate with hybrid approaches to develop custom tools for increased efficiency. As technology continues to advance, LLMs are poised to play an increasingly important role in abstract screening, reshaping the workflow of systematic review and meta-analysis studies."
W4410178347,,10.1186/s12929-025-01131-z,2025,Impact of large language model (ChatGPT) in healthcare: an umbrella review and evidence synthesis,Usman Iqbal,6,Journal of Biomedical Science,133,"BACKGROUND: The emergence of Artificial Intelligence (AI), particularly Chat Generative Pre-Trained Transformer (ChatGPT), a Large Language Model (LLM), in healthcare promises to reshape patient care, clinical decision-making, and medical education. This review aims to synthesise research findings to consolidate the implications of ChatGPT integration in healthcare and identify research gaps. MAIN BODY: The umbrella review was conducted following Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) guidelines. The Cochrane Library, PubMed, Scopus, Web of Science, and Google Scholar were searched from inception until February 2024. Due to the heterogeneity of the included studies, no quantitative analysis was performed. Instead, information was extracted, summarised, synthesised, and presented in a narrative form. Two reviewers undertook title, abstract, and full text screening independently. The methodological quality and overall rating of the included reviews were assessed using the A Measurement Tool to Assess systematic Reviews (AMSTAR-2) checklist. The review examined 17 studies, comprising 15 systematic reviews and 2 meta-analyses, on ChatGPT in healthcare, revealing diverse focuses. The AMSTAR-2 assessment identified 5 moderate and 12 low-quality reviews, with deficiencies like study design justification and funding source reporting. The most reported theme that emerged was ChatGPT's use in disease diagnosis or clinical decision-making. While 82.4% of studies focused on its general usage, 17.6% explored unique topics like its role in medical examinations and conducting systematic reviews. Among these, 52.9% targeted general healthcare, with 41.2% focusing on specific domains like radiology, neurosurgery, gastroenterology, public health dentistry, and ophthalmology. ChatGPT's use for manuscript review or writing was mentioned in 17.6% of reviews. Promising applications include enhancing patient care and clinical decision-making, though ethical, legal, and accuracy concerns require cautious integration. CONCLUSION: We summarise the identified areas in reviews regarding ChatGPT's transformative impact in healthcare, highlighting patient care, decision-making, and medical education. Emphasising the importance of ethical regulations and the involvement of policymakers, we urge further investigation to ensure the reliability of ChatGPT and to promote trust in healthcare and research."
W4392938433,,10.1007/s00432-024-05678-6,2024,Utilizing large language models in breast cancer management: systematic review,Vera Sorin,7,Journal of Cancer Research and Clinical Oncology,45,"PURPOSE: Despite advanced technologies in breast cancer management, challenges remain in efficiently interpreting vast clinical data for patient-specific insights. We reviewed the literature on how large language models (LLMs) such as ChatGPT might offer solutions in this field. METHODS: We searched MEDLINE for relevant studies published before December 22, 2023. Keywords included: ""large language models"", ""LLM"", ""GPT"", ""ChatGPT"", ""OpenAI"", and ""breast"". The risk bias was evaluated using the QUADAS-2 tool. RESULTS: Six studies evaluating either ChatGPT-3.5 or GPT-4, met our inclusion criteria. They explored clinical notes analysis, guideline-based question-answering, and patient management recommendations. Accuracy varied between studies, ranging from 50 to 98%. Higher accuracy was seen in structured tasks like information retrieval. Half of the studies used real patient data, adding practical clinical value. Challenges included inconsistent accuracy, dependency on the way questions are posed (prompt-dependency), and in some cases, missing critical clinical information. CONCLUSION: LLMs hold potential in breast cancer care, especially in textual information extraction and guideline-driven clinical question-answering. Yet, their inconsistent accuracy underscores the need for careful validation of these models, and the importance of ongoing supervision."
W4411329844,,10.1007/s12630-025-02973-9,2025,The applications of ChatGPT and other large language models in anesthesiology and critical care: a systematic review,Nicolas Daccache,4,Canadian Journal of Anesthesia/Journal canadien d anesthésie,15,
W4399707508,,10.1016/j.jval.2024.03.1461,2024,MSR28 Use of Large Language Model (LLM) for Full-Text Screening in Systematic Literature Reviews: A Comparative Analysis,H Rathi,4,Value in Health,2,
W4392964628,,10.1007/978-3-031-56027-9_25,2024,Zero-Shot Generative Large Language Models for Systematic Review Screening Automation,Shuai Wang,6,Lecture notes in computer science,23,
W4409341650,,10.1016/j.future.2025.107829,2025,The role of Large Language Models in addressing IoT challenges: A systematic literature review,Gabriele De Vito,3,Future Generation Computer Systems,20,"The Internet of Things (IoT) has revolutionized various sectors by enabling devices to communicate and interact seamlessly. However, developing IoT applications has data management, security, and interoperability challenges. Large Language Models (LLMs) have shown promise in addressing these challenges due to their advanced language processing capabilities. This Systematic Literature Review assesses the role of LLMs in addressing IoT challenges, exploring the strategies, hardware, and software configurations used, and identifying directions for future research. We extensively searched databases like Scopus, IEEE Xplore, and ACM Digital Library, initially screening 1419 studies and identifying an additional 1167 through snowballing, ultimately focusing on 55 relevant papers. The findings reveal LLMs’ potential to address key IoT challenges such as security and scalability. However, they also highlight significant obstacles, including high computational demands and the complexities of training and tuning these models. Future research should aim to develop methods to reduce the computational requirements of LLMs, improve training datasets, simplify implementation processes, and explore the ethical and privacy implications of using LLMs in IoT applications."
W4402141055,,10.1101/2024.09.01.24312887,2024,Large Language Models in Cardiology: A Systematic Review,Moran Gendler,8,medRxiv,5,"Background: Large language models (LLMs) are increasingly used in health care, but their role in cardiology has not yet been systematically evaluated. Objective: This review aimed to assess the applications, performance, and limitations of LLMs across diverse cardiology tasks, including chronic and progressive conditions, acute events, education, and diagnostic testing. Methods: A systematic search was conducted in PubMed and Scopus for studies published up to April 14, 2024, using keywords related to LLMs and cardiology. Studies evaluating LLM outputs in cardiology-related tasks were included. Data were extracted across 5 predefined domains and the risk of bias was assessed using an adapted QUADAS-2 tool (developed by Whiting et al at the University of Bristol). The review protocol was registered in PROSPERO (CRD42024556397). Results: A total of 33 studies contributed quantitative outcome data to a descriptive synthesis. Across chronic conditions, ChatGPT-3.5 (OpenAI) answered 91% (43/47) heart failure questions accurately, although readability often required college-level comprehension. In acute scenarios, Bing Chat omitted key myocardial infarction first aid steps in 25% (5/20) to 45% (9/20) of cases, while cardiac arrest information was rated highly (mean 4.3/5, SD 0.7) but written above recommended reading levels. In physician education tasks, ChatGPT-4 (OpenAI) demonstrated higher accuracy than ChatGPT-3.5, improving from 38% (33/88) to 66% (58/88). In patient education studies, ChatGPT-4 provided scientifically adequate explanations (5.0-6.0/7) comparable to hospital materials but at higher reading levels (11th vs 7th grade). In diagnostic testing, ChatGPT-4 interpreted 91% (36/40) electrocardiogram vignettes correctly, significantly better than emergency physicians (31/40, 77%; P< .001), but showed lower performance in echocardiography. Conclusions: LLMs show meaningful potential in cardiology, especially for education and electrocardiogram interpretation, but performance varies across clinical tasks. Limitations in emergency guidance and readability, as well as small in silico study designs, highlight the need for multimodal models and prospective validation."
W4391561379,,10.1038/s42256-023-00788-1,2024,Leveraging large language models for predictive chemistry,Kevin Maik Jablonka,4,Nature Machine Intelligence,377,"Abstract Machine learning has transformed many fields and has recently found applications in chemistry and materials science. The small datasets commonly found in chemistry sparked the development of sophisticated machine learning approaches that incorporate chemical knowledge for each application and, therefore, require specialized expertise to develop. Here we show that GPT-3, a large language model trained on vast amounts of text extracted from the Internet, can easily be adapted to solve various tasks in chemistry and materials science by fine-tuning it to answer chemical questions in natural language with the correct answer. We compared this approach with dedicated machine learning models for many applications spanning the properties of molecules and materials to the yield of chemical reactions. Surprisingly, our fine-tuned version of GPT-3 can perform comparably to or even outperform conventional machine learning techniques, in particular in the low-data limit. In addition, we can perform inverse design by simply inverting the questions. The ease of use and high performance, especially for small datasets, can impact the fundamental approach to using machine learning in the chemical and material sciences. In addition to a literature search, querying a pre-trained large language model might become a routine way to bootstrap a project by leveraging the collective knowledge encoded in these foundation models, or to provide a baseline for predictive tasks."
W7141008388,10.48550/arxiv.2603.24767,10.48550/arxiv.2603.24767,2026,Fine-Tuning A Large Language Model for Systematic Review Screening,Kweku Andoh Yamoah,5,arXiv (Cornell University),0,"Systematic reviews traditionally have taken considerable amounts of human time and energy to complete, in part due to the extensive number of titles and abstracts that must be reviewed for potential inclusion. Recently, researchers have begun to explore how to use large language models (LLMs) to make this process more efficient. However, research to date has shown inconsistent results. We posit this is because prompting alone may not provide sufficient context for the model(s) to perform well. In this study, we fine-tune a small 1.2 billion parameter open-weight LLM specifically for study screening in the context of a systematic review in which humans rated more than 8500 titles and abstracts for potential inclusion. Our results showed strong performance improvements from the fine-tuned model, with the weighted F1 score improving 80.79% compared to the base model. When run on the full dataset of 8,277 studies, the fine-tuned model had 86.40% agreement with the human coder, a 91.18% true positive rate, a 86.38% true negative rate, and perfect agreement across multiple inference runs. Taken together, our results show that there is promise for fine-tuning LLMs for title and abstract screening in large-scale systematic reviews."
W4404651681,,10.1007/s00701-024-06372-9,2024,Large language models in neurosurgery: a systematic review and meta-analysis,Advait Patil,6,Acta Neurochirurgica,26,
W7161451900,,,2026,Large Language Models for Web Accessibility: A Systematic Literature Review,Wajdi Aljedaani,2,arXiv (Cornell University),0,"Web accessibility aims to ensure that web content and services are usable by people with diverse abilities. In recent years, Large Language Models (LLMs) have been increasingly explored to support accessibility-related tasks on the web, such as content generation, issue detection, and remediation. However, little is known about the characteristics of these approaches, the accessibility issues they target, the standards they follow, and how they are evaluated. In this paper, we present a systematic literature review of 38 peer-reviewed studies that investigate the use of LLMs in web accessibility contexts. We begin by performing a comprehensive search of scientific publications to identify relevant studies. We then conduct a comparative analysis to examine the accessibility tasks addressed, the LLM models and prompting strategies employed, the system architectures adopted, the accessibility issues and guidelines considered, and the evaluation methods used across studies. Our findings show that most studies apply LLMs to text-centric and structurally explicit accessibility tasks, with WCAG serving as the primary reference framework and limited consideration of cognitive accessibility guidelines (COGA). The reviewed approaches predominantly rely on general-purpose LLMs and prompt-based interactions, while evaluation practices vary widely and often lack direct involvement of users with disabilities. We envision this review as a consolidated reference for researchers and practitioners seeking to understand the current landscape of LLM-supported web accessibility, and as a foundation to guide future research and tool development in this area."
W7142557847,,,2026,Fine-Tuning A Large Language Model for Systematic Review Screening,Kweku Andoh Yamoah,5,arXiv (Cornell University),0,"Systematic reviews traditionally have taken considerable amounts of human time and energy to complete, in part due to the extensive number of titles and abstracts that must be reviewed for potential inclusion. Recently, researchers have begun to explore how to use large language models (LLMs) to make this process more efficient. However, research to date has shown inconsistent results. We posit this is because prompting alone may not provide sufficient context for the model(s) to perform well. In this study, we fine-tune a small 1.2 billion parameter open-weight LLM specifically for study screening in the context of a systematic review in which humans rated more than 8500 titles and abstracts for potential inclusion. Our results showed strong performance improvements from the fine-tuned model, with the weighted F1 score improving 80.79% compared to the base model. When run on the full dataset of 8,277 studies, the fine-tuned model had 86.40% agreement with the human coder, a 91.18% true positive rate, a 86.38% true negative rate, and perfect agreement across multiple inference runs. Taken together, our results show that there is promise for fine-tuning LLMs for title and abstract screening in large-scale systematic reviews."
W7161293256,10.48550/arxiv.2605.13873,10.48550/arxiv.2605.13873,2026,Large Language Models for Web Accessibility: A Systematic Literature Review,Wajdi Aljedaani,2,arXiv (Cornell University),0,"Web accessibility aims to ensure that web content and services are usable by people with diverse abilities. In recent years, Large Language Models (LLMs) have been increasingly explored to support accessibility-related tasks on the web, such as content generation, issue detection, and remediation. However, little is known about the characteristics of these approaches, the accessibility issues they target, the standards they follow, and how they are evaluated. In this paper, we present a systematic literature review of 38 peer-reviewed studies that investigate the use of LLMs in web accessibility contexts. We begin by performing a comprehensive search of scientific publications to identify relevant studies. We then conduct a comparative analysis to examine the accessibility tasks addressed, the LLM models and prompting strategies employed, the system architectures adopted, the accessibility issues and guidelines considered, and the evaluation methods used across studies. Our findings show that most studies apply LLMs to text-centric and structurally explicit accessibility tasks, with WCAG serving as the primary reference framework and limited consideration of cognitive accessibility guidelines (COGA). The reviewed approaches predominantly rely on general-purpose LLMs and prompt-based interactions, while evaluation practices vary widely and often lack direct involvement of users with disabilities. We envision this review as a consolidated reference for researchers and practitioners seeking to understand the current landscape of LLM-supported web accessibility, and as a foundation to guide future research and tool development in this area."
W4412650425,,10.1093/ehjdh/ztaf028,2025,Applications of large language models in cardiovascular disease: a systematic review,José Ferreira Santos,4,European Heart Journal - Digital Health,33,"Cardiovascular disease (CVD) remains the leading cause of morbidity and mortality worldwide. Large language models (LLMs) offer potential solutions for enhancing patient education and supporting clinical decision-making. This study aimed to evaluate LLMs' applications in CVD and explore their current implementation, from prevention to treatment. Following the Preferred Reporting Items for Systematic Reviews and Meta-Analyses guidelines, this systematic review assessed LLM applications in CVD. A comprehensive PubMed search identified relevant studies. The review prioritized pragmatic and practical applications of LLMs. Key applications, benefits, and limitations of LLMs in CVD prevention were summarized. Thirty-five observational studies met the eligibility criteria. Of these, 54% addressed primary prevention and risk factor management, while 46% focused on established CVD. Commercial LLMs were evaluated in all but one study, with 91% (32 studies) assessing ChatGPT. The LLM applications were categorized as follows: 72% addressed patient education, 17% clinical decision support, and 11% both. In 68% of studies, the primary objective was to evaluate LLMs' performance in answering frequently asked patient questions, with results indicating accurate, comprehensive, and generally safe responses. However, occasional misinformation and hallucinated references were noted. Additional applications included patient guidance on CVD, first aid, and lifestyle recommendations. Large language models were assessed for medical questions, diagnostic support, and treatment recommendations in clinical decision support. Large language models hold significant potential in CVD prevention and treatment. Evidence supports their potential as an alternative source of information for addressing patients' questions about common CVD. However, further validation is needed for their application in individualized care, from diagnosis to treatment."
W7124830196,,10.2196/76734,2026,Large Language Models in Cardiology: Systematic Review,Moran Gendler,8,JMIR Cardio,3,"Abstract Background Large language models (LLMs) are increasingly used in health care, but their role in cardiology has not yet been systematically evaluated. Objective This review aimed to assess the applications, performance, and limitations of LLMs across diverse cardiology tasks, including chronic and progressive conditions, acute events, education, and diagnostic testing. Methods A systematic search was conducted in PubMed and Scopus for studies published up to April 14, 2024, using keywords related to LLMs and cardiology. Studies evaluating LLM outputs in cardiology-related tasks were included. Data were extracted across 5 predefined domains and the risk of bias was assessed using an adapted QUADAS-2 tool (developed by Whiting et al at the University of Bristol). The review protocol was registered in PROSPERO (CRD42024556397). Results A total of 33 studies contributed quantitative outcome data to a descriptive synthesis. Across chronic conditions, ChatGPT-3.5 (OpenAI) answered 91% (43/47) heart failure questions accurately, although readability often required college-level comprehension. In acute scenarios, Bing Chat omitted key myocardial infarction first aid steps in 25% (5/20) to 45% (9/20) of cases, while cardiac arrest information was rated highly (mean 4.3/5, SD 0.7) but written above recommended reading levels. In physician education tasks, ChatGPT-4 (OpenAI) demonstrated higher accuracy than ChatGPT-3.5, improving from 38% (33/88) to 66% (58/88). In patient education studies, ChatGPT-4 provided scientifically adequate explanations (5.0‐6.0/7) comparable to hospital materials but at higher reading levels (11th vs 7th grade). In diagnostic testing, ChatGPT-4 interpreted 91% (36/40) electrocardiogram vignettes correctly, significantly better than emergency physicians (31/40, 77%; P < .001), but showed lower performance in echocardiography. Conclusions LLMs show meaningful potential in cardiology, especially for education and electrocardiogram interpretation, but performance varies across clinical tasks. Limitations in emergency guidance and readability, as well as small in silico study designs, highlight the need for multimodal models and prospective validation."
W4417475506,,10.1038/s41366-025-01992-2,2025,Large language models in obesity: a systematic review,Thanathip Suenghataiphorn,4,International Journal of Obesity,6,
W4384641573,,10.1145/3539618.3591703,2023,Can ChatGPT Write a Good Boolean Query for Systematic Review Literature Search?,Shuai Wang,4,International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR),205,"Systematic reviews are comprehensive literature reviews for a highly focused research question. These reviews are considered the highest form of evidence in medicine. Complex Boolean queries are developed as part of the systematic review creation process to retrieve literature, as they permit reproducibility and understandability. However, it is difficult and time-consuming to develop high-quality Boolean queries, often requiring the expertise of expert searchers like librarians. Recent advances in transformer-based generative models have shown their ability to effectively follow user instructions and generate answers based on these instructions. In this paper, we investigate ChatGPT as a means for automatically formulating and refining complex Boolean queries for systematic review literature search. Overall, our research finds that ChatGPT has the potential to generate effective Boolean queries. The ability of ChatGPT to follow complex instructions and generate highly precise queries makes it a tool of potential value for researchers conducting systematic reviews, particularly for rapid reviews where time is a constraint and where one can trade off higher precision for lower recall. We also identify several caveats in using ChatGPT for this task, highlighting that this technology needs further validation before it is suitable for widespread uptake."
W4399052098,,10.1111/apt.18058,2024,Systematic review: The use of large language models as medical chatbots in digestive diseases,Mauro Giuffrè,7,Alimentary Pharmacology & Therapeutics,55,"BACKGROUND: Interest in large language models (LLMs), such as OpenAI's ChatGPT, across multiple specialties has grown as a source of patient-facing medical advice and provider-facing clinical decision support. The accuracy of LLM responses for gastroenterology and hepatology-related questions is unknown. AIMS: To evaluate the accuracy and potential safety implications for LLMs for the diagnosis, management and treatment of questions related to gastroenterology and hepatology. METHODS: We conducted a systematic literature search including Cochrane Library, Google Scholar, Ovid Embase, Ovid MEDLINE, PubMed, Scopus and the Web of Science Core Collection to identify relevant articles published from inception until January 28, 2024, using a combination of keywords and controlled vocabulary for LLMs and gastroenterology or hepatology. Accuracy was defined as the percentage of entirely correct answers. RESULTS: Among the 1671 reports screened, we identified 33 full-text articles on using LLMs in gastroenterology and hepatology and included 18 in the final analysis. The accuracy of question-responding varied across different model versions. For example, accuracy ranged from 6.4% to 45.5% with ChatGPT-3.5 and was between 40% and 91.4% with ChatGPT-4. In addition, the absence of standardised methodology and reporting metrics for studies involving LLMs places all the studies at a high risk of bias and does not allow for the generalisation of single-study results. CONCLUSIONS: Current general-purpose LLMs have unacceptably low accuracy on clinical gastroenterology and hepatology tasks, which may lead to adverse patient safety events through incorrect information or triage recommendations, which might overburden healthcare systems or delay necessary care."
W4404636419,,10.1016/j.jad.2024.11.052,2024,Exploring the efficacy and potential of large language models for depression: A systematic review,Mahmud Omar,2,Journal of Affective Disorders,52,
W4415344803,,10.1145/3801961,2026,Large Language Models for Combinatorial Optimization: A Systematic Review,Francesca Da Ros,4,ACM Computing Surveys,11,"This systematic review explores the application of Large Language Models (LLMs) in Combinatorial Optimization (CO). We report our findings using the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) guidelines. We conduct a literature search via Scopus and Google Scholar, examining over 2,000 publications. We assess publications against four inclusion and four exclusion criteria related to their language, research focus, publication year, and type. Eventually, we select 103 studies. We classify these studies into semantic categories and topics to provide a comprehensive overview of the field, including the tasks performed by LLMs, the architectures of LLMs, the existing datasets specifically designed for evaluating LLMs in CO, and the field of application. Finally, we identify future directions for leveraging LLMs in this field."
W7167724220,,10.59431/jda.v5i1.810,2026,Large Language Model Integration in Desktop Applications: A Systematic Literature Review,Deyidi Mokoginta,1,Journal Dekstop Application (JDA),0,"The integration of Large Language Models (LLMs) into desktop applications has accelerated sharply over the past several years, yet no prior systematic review has examined this deployment setting as a distinct research domain — separate from web-based or mobile environments, each carrying different architectural constraints, privacy expectations, and user interaction patterns. This study maps, analyzes, and synthesizes the available scientific evidence on LLM architectures adopted in desktop implementations, the technical barriers arising during integration, the measurable effects on User Experience (UX), and the relative effectiveness of competing deployment strategies. A Systematic Literature Review (SLR) was conducted following PRISMA 2020 reporting guidelines, with research questions structured using the PICOC framework. Systematic searches across seven digital databases — IEEE Xplore, ACM Digital Library, Scopus, Web of Science, Google Scholar, arXiv, and Semantic Scholar — covering 2019–2024 yielded 1,200 initial articles, narrowed to 42 final articles through layered selection and quality appraisal using a modified CASP rubric. GPT-4 and the LLaMA family emerged as the dominant architectures, with model selection driven primarily by parameter scale and privacy requirements. Computational constraints, response latency, and data security ranked as the most serious technical barriers, with Retrieval-Augmented Generation (RAG) and model quantization as the most common mitigations. LLM integration was associated with measurable gains in task efficiency and user satisfaction in 73.8% of reviewed studies, though output predictability and user trust remained persistent challenges. A hybrid deployment approach — combining local models with cloud-based API access — emerged as the most common practice among production implementations. Unlike existing reviews addressing LLM capabilities in general software engineering contexts, this study provides the first systematic map of LLM integration specifically within the desktop application domain, offering concrete guidance for developers working at the intersection of artificial intelligence and desktop software engineering."
W4407957917,,10.1016/j.jclinepi.2025.111746,2025,"Large language models for conducting systematic reviews: on the rise, but not yet ready for use—a scoping review",Judith-Lisa Lieberum,9,Journal of Clinical Epidemiology,146,"BACKGROUND AND OBJECTIVES: Machine learning promises versatile help in the creation of systematic reviews (SRs). Recently, further developments in the form of large language models (LLMs) and their application in SR conduct attracted attention. We aimed at providing an overview of LLM applications in SR conduct in health research. METHODS: We systematically searched MEDLINE, Web of Science, IEEEXplore, ACM Digital Library, Europe PMC (preprints), Google Scholar, and conducted an additional hand search (last search: February 26, 2024). We included scientific articles in English or German, published from April 2021 onwards, building upon the results of a mapping review that has not yet identified LLM applications to support SRs. Two reviewers independently screened studies for eligibility; after piloting, 1 reviewer extracted data, checked by another. RESULTS: Our database search yielded 8054 hits, and we identified 33 articles from our hand search. We finally included 37 articles on LLM support. LLM approaches covered 10 of 13 defined SR steps, most frequently literature search (n = 15, 41%), study selection (n = 14, 38%), and data extraction (n = 11, 30%). The mostly recurring LLM was Generative Pretrained Transformer (GPT) (n = 33, 89%). Validation studies were predominant (n = 21, 57%). In half of the studies, authors evaluated LLM use as promising (n = 20, 54%), one-quarter as neutral (n = 9, 24%) and one-fifth as nonpromising (n = 8, 22%). CONCLUSION: Although LLMs show promise in supporting SR creation, fully established or validated applications are often lacking. The rapid increase in research on LLMs for evidence synthesis production highlights their growing relevance. PLAIN LANGUAGE SUMMARY: Systematic reviews are a crucial tool in health research where experts carefully collect and analyze all available evidence on a specific research question. Creating these reviews is typically time- and resource-intensive, often taking months or even years to complete, as researchers must thoroughly search, evaluate, and synthesize an immense number of scientific studies. For the present article, we conducted a review to understand how new artificial intelligence (AI) tools, specifically large language models (LLMs) like Generative Pretrained Transformer (GPT), can be used to help create systematic reviews in health research. We searched multiple scientific databases and finally found 37 relevant articles. We found that LLMs have been tested to help with various parts of the systematic review process, particularly in 3 main areas: searching scientific literature (41% of studies), selecting relevant studies (38%), and extracting important information from these studies (30%). GPT was the most commonly used LLM, appearing in 89% of the studies. Most of the research (57%) focused on testing whether these AI tools actually work as intended in this context of systematic review production. The results were mixed: about half of the studies found LLMs promising, a quarter were neutral, and one-fifth found them not promising. While LLMs show potential for making the systematic review process more efficient, there is still a lack of fully tested and validated applications. However, the increasing number of studies in this field suggests that these AI tools are becoming increasingly important in creating systematic reviews."
W4399707564,,10.1016/j.jval.2024.03.1477,2024,MSR44 How Accurate Are Large Language Models for Abstract Screening in Systematic Literature Reviews?,A Sertel,5,Value in Health,1,
W4407135263,,10.1001/jamanetworkopen.2024.57879,2025,Large Language Models for Chatbot Health Advice Studies,Bright Huo,13,JAMA Network Open,279,"Importance: There is much interest in the clinical integration of large language models (LLMs) in health care. Many studies have assessed the ability of LLMs to provide health advice, but the quality of their reporting is uncertain. Objective: To perform a systematic review to examine the reporting variability among peer-reviewed studies evaluating the performance of generative artificial intelligence (AI)-driven chatbots for summarizing evidence and providing health advice to inform the development of the Chatbot Assessment Reporting Tool (CHART). Evidence Review: A search of MEDLINE via Ovid, Embase via Elsevier, and Web of Science from inception to October 27, 2023, was conducted with the help of a health sciences librarian to yield 7752 articles. Two reviewers screened articles by title and abstract followed by full-text review to identify primary studies evaluating the clinical accuracy of generative AI-driven chatbots in providing health advice (chatbot health advice studies). Two reviewers then performed data extraction for 137 eligible studies. Findings: A total of 137 studies were included. Studies examined topics in surgery (55 [40.1%]), medicine (51 [37.2%]), and primary care (13 [9.5%]). Many studies focused on treatment (91 [66.4%]), diagnosis (60 [43.8%]), or disease prevention (29 [21.2%]). Most studies (136 [99.3%]) evaluated inaccessible, closed-source LLMs and did not provide enough information to identify the version of the LLM under evaluation. All studies lacked a sufficient description of LLM characteristics, including temperature, token length, fine-tuning availability, layers, and other details. Most studies (136 [99.3%]) did not describe a prompt engineering phase in their study. The date of LLM querying was reported in 54 (39.4%) studies. Most studies (89 [65.0%]) used subjective means to define the successful performance of the chatbot, while less than one-third addressed the ethical, regulatory, and patient safety implications of the clinical integration of LLMs. Conclusions and Relevance: In this systematic review of 137 chatbot health advice studies, the reporting quality was heterogeneous and may inform the development of the CHART reporting standards. Ethical, regulatory, and patient safety considerations are crucial as interest grows in the clinical integration of LLMs."
W4412834208,,10.1016/j.infsof.2026.108240,2026,A systematic literature review on the use of large language models for optimization algorithms,Zoiya S. Morell,4,Information and Software Technology,1,"Context: Large language models (LLMs) are becoming popular in nearly every industry, with their problem-solving capabilities, ease of use, and interpretable results touted for complex optimization problems. However, LLMs used for optimization problems are prone to hallucinations and unstructured search techniques. Meanwhile, optimization algorithms (OA) have been used for such problems for years but are often designed for specific use cases and require domain expertise. This motivated a new combined field of LLM-modified OA.Objective: Recent studies on LLM-modified OA reveal promising results in improving optimization algorithms’ capabilities but demonstrate poor standardization, use of metrics, and repeatability. Method: In this paper, we surveyed research published from January 2017 to November 2024 to uncover how researchers are using LLMs and optimization algorithms in combination and review the quality of LLM-OA research. Following an extensive manual filtering procedure and critical analysis process, we evaluated 29 relevant papers merging LLMs and optimization algorithms to answer five research questions. Results: We found an absence of repeatability measures, statistical validity tests, and accountability for LLM hallucination in LLM-OA research. Since 68\% of papers do not provide adequate statistical analysis, 72\% lack proper metrics, and 86\% disregard hallucinations, we propose a standardized research methodology and highlight avenues for future research.Conclusions: LLMs for OAs remain largely unexplored, presenting several research opportunities."
W7116968648,,10.5335/rbca.v17i3.16310,2025,A Systematic Review about Large Language Models (LLMs) applied to Code Generation,Fabio Alecsandro Bacin,4,Revista Brasileira de Computação Aplicada,1,"Large Language Models (LLMs) for code generation represent a significant advancement in software development by boosting productivity, simplifying repetitive tasks, enabling automated testing, and promoting best practices. This paper presents a systematic literature review (SLR) of studies focused on LLMs applied to code generation. The review enhances understanding of the capabilities and limitations of these models, outlining both their benefits and challenges. The review protocol included a search on the Google Scholar database using relevant keywords related to LLMs and code generation. A total of 112 articles were initially retrieved, from which 15 were selected based on relevance and quality criteria. Of these, 8 were analyzed in depth to evaluate various approaches and outcomes, while the remaining 7 provided the theoretical foundation for the study. This work contributes to the growing body of knowledge in the field and supports future research and applications of LLMs in software engineering."
W4413110542,,10.1093/dmfr/twaf060,2025,The performance of large language models in dentomaxillofacial radiology: a systematic review,Zekai Liu,7,Dentomaxillofacial Radiology,26,"OBJECTIVES: This study aimed to systematically review the current performance of large language models (LLMs) in dento-maxillofacial radiology (DMFR). METHODS: Five electronic databases were used to identify studies that developed, fine-tuned, or evaluated LLMs for DMFR-related tasks. Data extracted included study purpose, LLM type, images/text source, applied language, dataset characteristics, input and output, performance outcomes, evaluation methods, and reference standards. Customized assessment criteria adapted from the TRIPOD-LLM reporting guideline were used to evaluate the risk-of-bias in the included studies specifically regarding the clarity of dataset origin, the robustness of performance evaluation methods, and the validity of the reference standards. RESULTS: The initial search yielded 1621 titles, and 19 studies were included. These studies investigated the use of LLMs for tasks including the production and answering of DMFR-related qualification exams and educational questions (n = 8), diagnosis and treatment recommendations (n = 7), and radiology report generation and patient communication (n = 4). LLMs demonstrated varied performance in diagnosing dental conditions, with accuracy ranging from 37% to 92.5% and expert ratings for differential diagnosis and treatment planning between 3.6 and 4.7 on a 5-point scale. For DMFR-related qualification exams and board-style questions, LLMs achieved correctness rates between 33.3% and 86.1%. Automated radiology report generation showed moderate performance with accuracy ranging from 70.4% to 81.3%. CONCLUSIONS: LLMs demonstrate promising potential in DMFR, particularly for diagnostic, educational, and report generation tasks. However, their current accuracy, completeness, and consistency remain variable. Further development, validation, and standardization are needed before LLMs can be reliably integrated as supportive tools in clinical workflows and educational settings."
W7134854403,,10.21203/rs.3.rs-9057643/v1,2026,Synthetic Participants Generated by Large Language Models: A Systematic Literature Review,Eduard Kuric,3,Research Square,2,
W7125646590,,10.5281/zenodo.18370396,2026,A Systematic Literature Review on the Use of Large Language Models for Optimization Algorithms,Zoiya S. Morell,4,Zenodo (CERN European Organization for Nuclear Research),0,"Context: Large language models (LLMs) are becoming popular in nearly every industry, with their problem-solving capabilities, ease of use, and interpretable results touted for complex optimization problems. However, LLMs used for optimization problems are prone to hallucinations and unstructured search techniques. Meanwhile, optimization algorithms (OA) have been used for such problems for years but are often designed for specific use cases and require domain expertise. This motivated a new combined field of LLM-modified OA. Objective: Recent studies on LLM-modified OA reveal promising results in improving optimization algorithms’ capabilities but demonstrate poor standardization, use of metrics, and repeatability. Method: In this paper, we surveyed research published from January 2017 to November 2024 to uncover how researchers are using LLMs and optimization algorithms in combination and review the quality of LLM-OA research. Following an extensive manual filtering procedure and critical analysis process, we evaluated 41 relevant papers merging LLMs and optimization algorithms to answer five research questions. Results: Although, the majority (62.2\%) of studies use LLMs during the search process (in-the-loop), we also found an absence of repeatability measures, statistical validity tests, and accountability for LLM hallucination in LLM-OA research. Since \pct{29} of papers do not provide adequate statistical analysis and \pct{30} disregard hallucinations, we propose a standardized research methodology and highlight avenues for future research. Conclusions: LLMs for OAs remain largely unexplored, presenting several research opportunities."
W4414751697,,10.2196/74177,2025,Large Language Models in Lung Cancer: Systematic Review,Ruikang Zhong,9,Journal of Medical Internet Research,9,"Background: In the era of data and intelligence, artificial intelligence has been widely applied in the medical field. As the most cutting-edge technology, the large language model (LLM) has gained popularity due to its extraordinary ability to handle complex tasks and interactive features. Objective: This study aimed to systematically review current applications of LLMs in lung cancer (LC) care and evaluate their potential across the full-cycle management spectrum. Methods: Following PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) guidelines, we conducted a comprehensive literature search across 6 databases up to January 1, 2025. Studies were included if they satisfied the following criteria: (1) journal articles, conference papers, and preprints; (2) studies that reported the content of LLMs in LC; (3) including original data and LC-related data presented separately; and (4) studies published in English. The exclusion criteria were as follows: (1) books and book chapters, letters, reviews, conference proceedings; (2) studies that did not report the content of LLMs in LC; and (3) no original data, and LC-related data that are not presented separately. Studies were screened independently by 2 authors (SC and ZL) and assessed for quality using Quality Assessment of Diagnostic Accuracy Studies-2, Prediction Model Risk of Bias Assessment Tool, and Risk Of Bias in Non-randomized Studies - of Interventions tools, selected based on study type. Key data items extracted included model type, application scenario, prompt method, input and output format, outcome measures, and safety considerations. Data analysis was conducted using descriptive statistics. Results: Out of 706 studies screened, 28 were included (published between 2023 and 2024). The ability of LLMs to automatically extract medical records, popularize general knowledge about LC, and assist clinical diagnosis and treatment has been demonstrated through the systematic review, emerging visual ability, and multimodal potential. Prompt engineering was a critical component, with varying degrees of sophistication from zero-shot to fine-tuned approaches. Quality assessments revealed overall acceptable methodological rigor but noted limitations in bias control and data security reporting. Conclusions: LLMs show considerable potential in improving LC diagnosis, communication, and decision-making. However, their responsible use requires attention to privacy, interpretability, and human oversight."
W7171869200,,10.14744/hf.2026.65328,2026,Large Language Models in Hepatology: A Systematic Review,Thanathip Suenghataiphorn,4,Hepatology forum/Hepatology forum (Online),0,"Background and Aim: The rapid advancement of generative artificial intelligence (AI), particularly large language models (LLMs), has opened new frontiers in healthcare, with emerging implications for hepatology. This systematic review synthesizes the current state of research on the application of LLMs in hepatology, focusing on their capabilities in real-world clinical settings, limitations, and future directions. Materials and Methods: Electronic databases, including MEDLINE, EMBASE, and OVID as a search platform, were used to identify eligible studies from inception to January 2025. Eligible studies investigated the clinical utility and performance of LLMs in hepatology, with a clear comparison to a defined ground truth. Key findings were extracted and synthesized narratively. The ROBINS-I tool was used to assess the risk of bias in each study. Results: Twenty-one studies were included in this review. Our analysis reveals that LLMs demonstrate promising capabilities in processing textual and visual data related to various liver diseases, including hepatocellular carcinoma, cirrhosis, and non-alcoholic fatty liver disease. LLMs effectively assisted with radiological image interpretation, provided clinical decision support, and generated patient education materials. However, the accuracy of these models was highly variable, depending on the specific task and the complexity of the clinical scenario. Limitations, such as the generation of inaccurate or misleading information (""hallucinations""), dependence on training data quality, and ethical considerations, were identified across multiple studies. Conclusion: Generative AI demonstrates feasibility across various hepatology applications, but study heterogeneity and significant challenges remain regarding accuracy, reliability, and safety. Future integration necessitates further research into training methods, data quality, ethical considerations, and real-world validation against standardized benchmarks."
W7106340049,,10.71781/25708,2025,Automatic screening of articles for systematic reviews using large language models,"Kumar, Gauransh",1,,0,"Les revues systématiques de la littérature sont essentielles pour la recherche fondée sur des preuves mais exigent un travail manuel considérable, en particulier lors du tri des articles pour en déterminer la pertinence. Cette thèse explore comment les grands modèles de langage (Large Language Models, LLMs) peuvent aider à réduire cette charge tout en maintenant la qualité scientifique. Nous avons testé les LLMs comme assistants de sélection à laide de jeux de données variés et examiné limpact des différentes conceptions dinvite sur les performances. Notre étude a analysé la généralisabilité des invites entre différentes familles de LLMs, la relation entre la confiance du modèle et sa précision, ainsi que des stratégies pour combiner plusieurs approches. Nous avons développé PromptSLR, un outil permettant des tests contrôlés dinvites et une analyse de performance. Grâce à lélimination systématique des caractéristiques des invites, nous avons identifié des conceptions optimales à laide danalyses statistiques. Les résultats montrent que 2 invites atteignent les meilleures performances sur un ensemble de près de 30 invites, mais que leur efficacité varie fortement selon les modèles de LLM, avec une transférabilité limitée entre modèles. De façon critique, nous avons constaté que les LLMs excellent davantage à identifier les articles non pertinents quà sélectionner les pertinents, ce qui les rend idéaux pour le pré-tri plutôt que pour le tri principal. Il est observé que demander aux LLMs de sauto-évaluer (score de confiance) améliore leurs performances. Une confiance plus élevée du modèle corrèle avec une meilleure précision, permettant aux chercheurs de concentrer leffort manuel sur les cas incertains tout en automatisant lexclusion des évidences. Nous proposons une approche hybride combinant le pré-tri par LLM et le tri manuel traditionnel, ce qui réduit substantiellement la charge de travail des réviseurs tout en préservant la qualité du processus. Cette thèse positionne les LLMs comme des outils dassistance qui complètent lexpertise humaine plutôt que de la remplacer, et contribue à fournir des méthodes pratiques pour améliorer l'efficacité des SLRs ainsi que des outils pour une mise en oeuvre réelle."
W4411569746,,10.1017/rsm.2025.10014,2025,Optimal large language models to screen citations for systematic reviews,Takehiko Oami,3,Research Synthesis Methods,16,"Abstract Recent studies highlight the potential of large language models (LLMs) in citation screening for systematic reviews; however, the efficiency of individual LLMs for this application remains unclear. This study aimed to compare accuracy, time-related efficiency, cost, and consistency across four LLMs—GPT-4o, Gemini 1.5 Pro, Claude 3.5 Sonnet, and Llama 3.3 70B—for literature screening tasks. The models screened for clinical questions from the Japanese Clinical Practice Guidelines for the Management of Sepsis and Septic Shock 2024. Sensitivity and specificity were calculated for each model based on conventional citation screening results for qualitative assessment. We also recorded the time and cost of screening and assessed consistency to verify reproducibility. A post hoc analysis explored whether integrating outputs from multiple models could enhance screening accuracy. GPT-4o and Llama 3.3 70B achieved high specificity but lower sensitivity, while Gemini 1.5 Pro and Claude 3.5 Sonnet exhibited higher sensitivity at the cost of lower specificity. Citation screening times and costs varied, with GPT-4o being the fastest and Llama 3.3 70B the most cost-effective. Consistency was comparable among the models. An ensemble approach combining model outputs improved sensitivity but increased the number of false positives, requiring additional review effort. Each model demonstrated distinct strengths, effectively streamlining citation screening by saving time and reducing workload. However, reviewing false positives remains a challenge. Combining models may enhance sensitivity, indicating the potential of LLMs to optimize systematic review workflows."
W7201993829,,10.1111/jocn.70490,2026,ChatGPT and Large Language Models in Contemporary Nursing,Izabella Uchmanowicz,10,Journal of Clinical Nursing,0,"OBJECTIVES: This narrative review synthesizes published evidence on the applications, benefits, limitations and governance considerations of ChatGPT and large language models (LLMs) in nursing, across three domains: education, clinical practice and workflow management. DESIGN: The article was conducted as a narrative review. METHODS: A structured literature search was conducted in Medline (via PubMed), Scopus and arXiv, covering publications from January 2019 to March 2026. Peer-reviewed original studies, systematic reviews, scoping reviews, narrative reviews and expert commentaries addressing LLM applications in nursing education, clinical practice or workflow were eligible for inclusion. Studies limited exclusively to non-nursing medical specialties without transferable nursing implications were excluded. Findings were narratively synthesized across five thematic domains by authors with subject-matter expertise in each area. RESULTS: In nursing education, ChatGPT demonstrates utility as an adaptive cognitive scaffold, supporting theoretical learning, simulation-based training and virtual patient encounters, though unregulated use poses risks to academic integrity and independent clinical reasoning. In clinical practice, LLMs can assist with preliminary symptom assessment and patient education material generation; however, performance deteriorates markedly in complex or data-sparse clinical scenarios and hallucination rates remain clinically significant. In workflow management, ChatGPT shows promise in reducing documentation burden and supporting administrative tasks, though data privacy obligations under frameworks such as GDPR constrain real-world deployment. Across all domains, concerns persist regarding algorithmic bias, professional accountability and the absence of clear medico-legal frameworks governing AI-related clinical errors. CONCLUSION: ChatGPT and related LLMs are best positioned as auxiliary tools that augment rather than replace professional nursing judgement. Safe and ethical integration requires the development of AI literacy curricula, institutionally governed deployment frameworks, mandatory human-in-the-loop verification protocols and longitudinal evaluation of patient safety outcomes. Nurses must play an active role in shaping the responsible adoption of generative AI in healthcare. IMPLICATIONS FOR PRACTICE AND RESEARCH: 1. Nurses must treat AI-generated content as a preliminary draft requiring mandatory human verification before clinical or documentation use. 2. Institutions should prioritize closed-loop, enterprise-grade AI deployments over public platforms to ensure GDPR compliance. 3. AI literacy must be embedded in undergraduate and continuing nursing education curricula. 4. Longitudinal research on patient safety outcomes following real-world LLM deployment in nursing is urgently needed. REPORTING METHOD: As a narrative review, this article followed established guidance for the conduct and reporting of narrative reviews. PATIENT OR PUBLIC CONTRIBUTION: There was no patient or public involvement in this narrative review."
W4404288320,,10.1016/j.identj.2024.10.014,2024,Large Language Models in Dental Licensing Examinations: Systematic Review and Meta-Analysis,Mingxin Liu,7,International Dental Journal,41,"INTRODUCTION AND AIMS: This study systematically reviews and conducts a meta-analysis to evaluate the performance of various large language models (LLMs) in dental licensing examinations worldwide. The aim is to assess the accuracy of these models in different linguistic and geographical contexts. This will inform their potential application in dental education and diagnostics. METHODS: Following Preferred Reporting Items for Systematic Reviews and Meta-Analyses guidelines, we conducted a comprehensive search across PubMed, Web of Science, and Scopus for studies published from 1 January 2022 to 1 May 2024. Two authors independently reviewed the literature based on the inclusion and exclusion criteria, extracted data, and evaluated the quality of the studies in accordance with the Quality Assessment of Diagnostic Accuracy Studies-2. We conducted qualitative and quantitative analyses to evaluate the performance of LLMs. RESULTS: Eleven studies met the inclusion criteria, encompassing dental licensing examinations from eight countries. GPT-3.5, GPT-4, and Bard achieved integrated accuracy rates of 54%, 72%, and 56%, respectively. GPT-4 outperformed GPT-3.5 and Bard, passing more than half of the dental licensing examinations. Subgroup analyses and meta-regression showed that GPT-3.5 performed significantly better in English-speaking countries. GPT-4's performance, however, remained consistent across different regions. CONCLUSION: LLMs, particularly GPT-4, show potential in dental education and diagnostics, yet their accuracy remains below the threshold required for clinical application. The lack of sufficient training data in dentistry has affected LLMs' accuracy. The reliance on image-based diagnostics also presents challenges. As a result, their accuracy in dental exams is lower compared to medical licensing exams. Additionally, LLMs even provide more detailed explanation for incorrect answer than correct one. Overall, the current LLMs are not yet suitable for use in dental education and clinical diagnosis."
W4413422599,,10.1016/j.acra.2025.08.014,2025,Evaluating Large Language Models for Radiology Systematic Review Title and Abstract Screening,Siddhant Dogra,9,Academic Radiology,6,
W4414801921,,10.1016/j.jclinane.2025.112030,2025,Combining large language models enhances screening sensitivity in systematic reviews,Alessandro DE CASSAI,6,Journal of Clinical Anesthesia,3,
W4408801716,,10.1017/rsm.2025.15,2025,Validation of large language models (Llama 3 and ChatGPT-4o mini) for title and abstract screening in biomedical systematic reviews,Adriana López‐Pineda,6,Research Synthesis Methods,17,"With the increasing volume of scientific literature, there is a need to streamline the screening process for titles and abstracts in systematic reviews, reduce the workload for reviewers, and minimize errors. This study validated artificial intelligence (AI) tools, specifically Llama 3 70B via Groq's application programming interface (API) and ChatGPT-4o mini via OpenAI's API, for automating this process in biomedical research. It compared these AI tools with human reviewers using 1,081 articles after duplicate removal. Each AI model was tested in three configurations to assess sensitivity, specificity, predictive values, and likelihood ratios. The Llama 3 model's LLA_2 configuration achieved 77.5% sensitivity and 91.4% specificity, with 90.2% accuracy, a positive predictive value (PPV) of 44.3%, and a negative predictive value (NPV) of 97.9%. The ChatGPT-4o mini model's CHAT_2 configuration showed 56.2% sensitivity, 95.1% specificity, 92.0% accuracy, a PPV of 50.6%, and an NPV of 96.1%. Both models demonstrated strong specificity, with CHAT_2 having higher overall accuracy. Despite these promising results, manual validation remains necessary to address false positives and negatives, ensuring that no important studies are overlooked. This study suggests that AI can significantly enhance efficiency and accuracy in systematic reviews, potentially revolutionizing not only biomedical research but also other fields requiring extensive literature reviews."
W4408763854,,10.1093/jamia/ocaf050,2025,High-performance automated abstract screening with large language model ensembles,Rohan Sanghera,10,Journal of the American Medical Informatics Association,55,"OBJECTIVE: screening is a labor-intensive component of systematic review involving repetitive application of inclusion and exclusion criteria on a large volume of studies. We aimed to validate large language models (LLMs) used to automate abstract screening. MATERIALS AND METHODS: LLMs (GPT-3.5 Turbo, GPT-4 Turbo, GPT-4o, Llama 3 70B, Gemini 1.5 Pro, and Claude Sonnet 3.5) were trialed across 23 Cochrane Library systematic reviews to evaluate their accuracy in zero-shot binary classification for abstract screening. Initial evaluation on a balanced development dataset (n = 800) identified optimal prompting strategies, and the best performing LLM-prompt combinations were then validated on a comprehensive dataset of replicated search results (n = 119 695). RESULTS: On the development dataset, LLMs exhibited superior performance to human researchers in terms of sensitivity (LLMmax = 1.000, humanmax = 0.775), precision (LLMmax = 0.927, humanmax = 0.911), and balanced accuracy (LLMmax = 0.904, humanmax = 0.865). When evaluated on the comprehensive dataset, the best performing LLM-prompt combinations exhibited consistent sensitivity (range 0.756-1.000) but diminished precision (range 0.004-0.096) due to class imbalance. In addition, 66 LLM-human and LLM-LLM ensembles exhibited perfect sensitivity with a maximal precision of 0.458 with the development dataset, decreasing to 0.1450 over the comprehensive dataset; but conferring workload reductions ranging between 37.55% and 99.11%. DISCUSSION: Automated abstract screening can reduce the screening workload in systematic review while maintaining quality. Performance variation between reviews highlights the importance of domain-specific validation before autonomous deployment. LLM-human ensembles can achieve similar benefits while maintaining human oversight over all records. CONCLUSION: LLMs may reduce the human labor cost of systematic review with maintained or improved accuracy, thereby increasing the efficiency and quality of evidence synthesis."
W7213653222,,10.17605/osf.io/2k3zj,2026,Large language models for title and abstract screening in oncology systematic reviews,Tao Xu,2,Open Science Framework,0,"This is a retrospective registration of a completed diagnostic accuracy study. Analyses had already been conducted at the time of registration. Purpose: Title–abstract screening is a major source of workload in systematic reviews. Missing an eligible randomised trial can change the included evidence, whereas retaining an ineligible record only adds later reading. This project evaluates a predefined first-pass workflow: one frozen recall-first prompt, five commercially available large language models, and human reading limited to records labelled include or uncertain. Methods: The index test was applied to the original search yields of 45 published oncology randomised-trial meta-analyses (128,023 unique records; 611 unique reference-standard inclusions). The reference standard was each source review’s final included set after full-text assessment. Recall was scored on 641 reference-standard inclusions with abstracts that were present in all five model tables. Models were DeepSeek-chat, Qwen-Plus, Gemini 2.5 Flash, GPT-4.1 mini, and GPT-4o (temperature 0). Uncertain was counted with include. Prompt templates were compared on a locked 800-record test set after a disjoint 312-record development set; the production prompt was not changed by that catalogue. Outcomes: The primary diagnostic accuracy outcome is recall of reference-standard inclusions with abstracts. The co-primary process measure is workload reduction, 1 − (records retained for human screening / N). Secondary measures include precision, number needed to read, estimated reading time at 30 seconds per record, and the number of published inclusions that would remain unread if excluded records were not reassessed."
W4414661911,,10.3389/fdgth.2025.1659134,2025,Large language models in real-world clinical workflows: a systematic review of applications and implementation,Yaara Artsi,6,Frontiers in Digital Health,84,"Background: Large language models (LLMs) offer promise for enhancing clinical care by automating documentation, supporting decision-making, and improving communication. However, their integration into real-world healthcare workflows remains limited and under characterized. This systematic review aims to evaluate the literature on real-world implementation of LLMs in clinical workflows, including their use cases, clinical settings, observed outcomes, and challenges. Methods: We searched MEDLINE, Scopus, Web of Science, and Google Scholar for studies published between January 2015 and April 2025 that assessed LLMs in real-world clinical applications. Inclusion criteria were peer-reviewed, full-text studies in English reporting empirical implementation of LLMs in clinical settings. Study quality and risk of bias were assessed using the PROBAST tool. Results: Four studies published between 2024 and 2025 met inclusion criteria. All used generative pre-trained transformers (GPTs). Reported applications included outpatient communication, mental health support, inbox message drafting, and clinical data extraction. LLM deployment was associated with improvements in operational efficiency, user satisfaction, and reduced workload. However, challenges included performance variability across data types, limitations in generalizability, regulatory delays, and lack of post-deployment monitoring. Conclusions: Early evidence suggests that LLMs can enhance clinical workflows, but real-world adoption remains constrained by systemic, technical, and regulatory barriers. To support safe and scalable use, future efforts should prioritize standardized evaluation metrics, multi-site validation, human oversight, and implementation frameworks tailored to clinical settings. Systematic Review Registration: https://www.crd.york.ac.uk/PROSPERO/recorddashboard, PROSPERO CRD420251030069."
W7135053800,,10.1007/s42600-026-00465-3,2026,Large language models in portuguese for healthcare: a systematic review,Andre Massahiro Shimaoka,7,Research on Biomedical Engineering,1,"Abstract Purpose This study addresses Large Language Models (LLMs) pre-trained in Portuguese for healthcare applications, focusing on contextual embeddings. Research on LLMs for natural language processing (NLP) tasks in Portuguese is limited, especially within healthcare. Much of the existing research has focused on high-resource languages such as English. However, LLMs demonstrate potential in clinical decision support, diagnosis assistance, patient care, and other healthcare applications. In view of this, the present work assesses the current state of LLMs in Portuguese for healthcare. Methods Our Systematic Literature Review (SLR) followed standard protocols: search, screening based on inclusion/exclusion criteria, quality assessment, data extraction, and analysis. Results We identified 32 models, mostly based on BERTimbau, mBERT, and BioBERTpt. Adaptation strategies such as fine-tuning, domain-adaptive pre-training, training from scratch, and zero-shot learning have been the most prevalent. Several datasets have been used, including clinical records, social media, and scientific repositories. LLMs in Portuguese are being applied in mental health, general medicine, COVID-19, oncology, and other related areas, accomplishing classification tasks, followed by named entity recognition (NER), topic modeling, question answering, text generation, summarization, de-identification, and conversational agents. Conclusion Our study identified key gaps and opportunities: (1) unexplored recent LLMs such as T5, Qwen, DeepSeek, BART, among others; (2) insufficient fine-tuning details, hindering reproducibility; (3) limited coverage of healthcare fields; (4) clinical and hospital data widely used but not shared; (5) social media data requiring caution due to potential inconsistencies; and (6) overlooked data privacy."
W4390700469,,10.1101/2024.01.05.24300884,2024,Large language models in simplifying radiological reports: systematic review,Yaara Artsi,6,medRxiv,9,"Abstract Objectives Simplifying medical information to make it understandable for patients, specifically in the case of radiology reports, is challenging. It requires time and effort from medical personnel. This systematic review focuses on the application of large language models (LLMs) in generating simplified radiological imaging reports, as well as answering patient inquiries regarding radiological procedures. Materials and Methods The authors searched for studies published up to January 2024. Search terms focused on LLMs generated simplified radiological reports and answers to patient inquiries regarding radiological procedures. MEDLINE was used as a search database. Results Overall, eight studies published between May 2023 and November 2023 were included. All studies showed that LLMs can produce simplified medical information for patients. Four studies (50%) used GPT-3.5, Two studies (25%) conducted a comparative analysis between GPT-3.5 and GPT-4. One study (12.5%) examined Microsoft Bing. One study (12.5%) utilized GPT-4. Four studies (50%) used LLMs to simplify radiological reports. Four studies (50%) used LLMs to answer patient questions regarding radiological procedures. Only two studies (25%) used patients to evaluate the LLMs output. One study (12.5%) compared their initial prompt with optimized prompt. Five studies (62.5%) showed missing, inaccurate and potentially harmful AI outputs. Conclusion LLMs can be used to simplify medical imaging reports and procedures, for improved patient comprehension. However, their limitations cannot be ignored. Further study in this field is essential and more conclusive evidence is needed."
W4390897455,10.48550/arxiv.2401.06320,10.48550/arxiv.2401.06320,2024,Zero-shot Generative Large Language Models for Systematic Review Screening Automation,Shuai Wang,6,arXiv (Cornell University),5,"Systematic reviews are crucial for evidence-based medicine as they comprehensively analyse published research findings on specific questions. Conducting such reviews is often resource- and time-intensive, especially in the screening phase, where abstracts of publications are assessed for inclusion in a review. This study investigates the effectiveness of using zero-shot large language models~(LLMs) for automatic screening. We evaluate the effectiveness of eight different LLMs and investigate a calibration technique that uses a predefined recall threshold to determine whether a publication should be included in a systematic review. Our comprehensive evaluation using five standard test collections shows that instruction fine-tuning plays an important role in screening, that calibration renders LLMs practical for achieving a targeted recall, and that combining both with an ensemble of zero-shot models saves significant screening time compared to state-of-the-art approaches."
W7132993116,,,2025,Prompting Strategies for Large Language Models in Requirements Elicitation : A Systematic Literature Review,Naveed Ali,1,Tampere University Institutional Repository (Tampere University),0,"Recent advancements in Large Language Models (LLMs) such as GPT-4, DeepSeek, Gemini, and LLaMA have introduced new opportunities for supporting requirements elicitation, one of the most knowledge-intensive and error-prone activities in Requirements Engineering (RE). The effectiveness of LLM-assisted elicitation depends largely on how prompts are designed, structured, and contextualized. As a wide range of prompting strategies has emerged, including zero-shot, few-shot, chain-of-thought, role-based, and prompt-chaining techniques, there is a growing need to consolidate current evidence and evaluate their strengths, limitations, and practical relevance. This thesis presents a systematic literature review of 58 peer-reviewed studies investigating prompting strategies used with LLMs for requirements elicitation. The review follows established SLR guidelines and PRISMA 2020 procedures. Studies were retrieved from Scopus, IEEE Xplore, and the ACM Digital Library using a unified search strategy and underwent a structured screening, quality assessment, and data extraction process. The findings categorize prompting strategies across multiple dimensions, including their structural design, reasoning mechanisms, and integration with supporting tools such as retrieval-augmented generation pipelines, conversational agents, and domain-specific knowledge models. The review also synthesizes how prompting is applied across key elicitation contexts. Furthermore, it examines evaluation approaches, highlighting trends in quantitative metrics, expert judgment, and mixed-method assessment. The study contributes a consolidated view of prompting practices in LLM-based elicitation, identifies recurring benefits and challenges, and outlines emerging best practices. By synthesizing methodological patterns and gaps, this review offers guidance for researchers and practitioners seeking to design effective prompting strategies and develop reliable, human-centered elicitation workflows supported by generative AI."
W4407395599,,10.1097/hm9.0000000000000143,2025,Large language models in traditional Chinese medicine: a systematic review,Zhe Chen,8,Acupuncture and Herbal Medicine,16,"Objective: Generative artificial intelligence (AI) technology, represented by large language models (LLMs), has gradually been developed for traditional Chinese medicine (TCM); however, challenges remain in effectively enhancing AI applications for TCM. Therefore, this study is the first systematic review to analyze LLMs in TCM retrospectively, focusing on and summarizing the evidence of their performance in generative tasks. Methods: We extensively searched electronic databases for articles published until June 2024 to identify publicly available studies on LLMs in TCM. Two investigators independently selected and extracted the related information and evaluation metrics. Based on the available data, this study used descriptive analysis for a comprehensive systematic review of LLM technology related to TCM. Results: Ten studies published between 2023 and 2024 met our eligibility criteria and were included in this review, including 40% LLMs in the TCM vertical domain, 40% containing TCM data, and 20% honoring the TCM contribution, with a foundational model parameter range from 1.8 to 33 billion. All included studies used manual or automatic evaluation metrics to evaluate model performance and fully discussed the challenges and contributions through an overview of LLMs in TCM. Conclusions: LLMs have achieved significant advantages in TCM applications and can effectively address intelligent TCM tasks. Further in-depth development of LLMs is needed in various vertical TCM fields, including clinical and fundamental research. Focusing on the functional segmentation development direction of generative AI technologies in TCM application scenarios to meet the practical needs-oriented demands of TCM digitalization is essential."
W7156901103,,10.1002/widm.70089,2026,Large Language Models for Explainable Medical Text Summarization: A Systematic Literature Review,Aleka Melese Ayalew,4,Wiley Interdisciplinary Reviews Data Mining and Knowledge Discovery,3,"ABSTRACT Recently, large language models have gained momentum in several areas, including medical text summarization and clinical decision support in healthcare. In this systematic review, we focus on how LLMs are used for summarization and explainability, and on how they improve the precision of clinical decisions. Initially, by screening eight distinct databases, we identified 1601 studies using the PRISMA protocol. Following strict qualifying criteria, 61 studies were selected for the final review. Our results show that decoder‐only architectures dominate the field (60.7% of studies), with zero‐shot prompting emerging as the predominant approach (55.7%), while 24.6% of studies employed fine‐tuning. General clinical decision support (26.2%) and diagnostic assistance (19.7%) constitute the two most widely used clinical applications across multiple clinical specialties. Identified in 26.2% of studies, the intersection of summarization and explainability becomes a key focus area, with fine‐tuned models obtaining 38.6% higher average ROUGE‐1 scores compared to zero‐shot approaches, and studies including robust explainability features report 27.3% higher clinician acceptance rates. Despite great potential, major obstacles still exist, including model hallucinations (63.9% of studies), minimal workflow integration (only 14.8% of studies), and inadequate attention to regulatory paths (8.2%). Although LLMs offer significant potential for transforming clinical decision support via improved information handling and clear reasoning, realizing these goals requires overcoming essential challenges in clinical validation, system integration, and ethical oversight. This review aims to provide an extensive framework for understanding the current capabilities, limitations, and prospects of LLM‐based explainable summarization in clinical settings, guiding future research and practical deployment. This article is categorized under: Technologies > Artificial Intelligence Fundamental Concepts of Data and Knowledge > Explainable AI Application Areas > Health Care"
W4410090731,,10.2196/preprints.76734,2025,Large Language Models in Cardiology: A Systematic Review (Preprint),Moran Gendler,8,,0,"UNSTRUCTURED Purpose: This review analyzes the application of large language models (LLMs) in the field of cardiology, with a focus on evaluating their performances across various clinical tasks. Methods: We conducted a systematic literature search on PubMed for studies published up to April 14, 2024. Our search strategy involved a combination of multiple keywords pertaining to LLMs and various aspects of cardiology. The risk of bias was evaluated using the QUADAS-2 tool. Results: Fifteen studies met the inclusion criteria, categorized into four domains: chronic and progressive cardiac conditions, acute cardiac events, cardiology education, and cardiac monitoring. Six studies addressing chronic conditions demonstrated variability in the accuracy and depth of LLM-generated responses. In acute cardiac scenarios, three articles showed that LLMs provided medical advice with mixed effectiveness, particularly in delivering CPR instructions. Two studies in educational cardiology revealed high accuracy in answering assessment questions and interpreting clinical cases. Finally, four articles on cardiac diagnostics showed that multimodal LLMs displayed a range of capabilities in ECGs interpretation, with some models performing at or exceeding the level of human specialists. Conclusion: LLMs demonstrate considerable potential in the field of cardiology, particularly in educational applications and routine diagnostics. However, their performance remains inconsistent across various clinical scenarios, particularly in acute care settings where precision is critical. Enhancing LLMs' accuracy in interpreting real-world complex medical data and emergency response guidance is imperative before integration into clinical practice."
W4409970142,,10.7717/peerj-cs.2822,2025,Validation of automated paper screening for esophagectomy systematic review using large language models,Rashi Ramchandani,10,PeerJ Computer Science,5,"Background: Large language models (LLMs) offer a potential solution to the labor-intensive nature of systematic reviews. This study evaluated the ability of the GPT model to identify articles that discuss perioperative risk factors for esophagectomy complications. To test the performance of the model, we tested GPT-4 on narrower inclusion criterion and by assessing its ability to discriminate relevant articles that solely identified preoperative risk factors for esophagectomy. Methods: = 1,967) discussing risk factors to esophagectomy complications. The articles underwent title and abstract screening by three independent human reviewers and GPT-4. The Python script used for the analysis made Application Programming Interface (API) calls to GPT-4 with screening criteria in natural language. GPT-4's inclusion and exclusion decision were compared to those decided human reviewers. Results: The agreement between the GPT model and human decision was 85.58% for perioperative factors and 78.75% for preoperative factors. The AUC value was 0.87 and 0.75 for the perioperative and preoperative risk factors query, respectively. In the evaluation of perioperative risk factors, the GPT model demonstrated a high recall for included studies at 89%, a positive predictive value of 74%, and a negative predictive value of 84%, with a low false positive rate of 6% and a macro-F1 score of 0.81. For preoperative risk factors, the model showed a recall of 67% for included studies, a positive predictive value of 65%, and a negative predictive value of 85%, with a false positive rate of 15% and a macro-F1 score of 0.66. The interobserver reliability was substantial, with a kappa score of 0.69 for perioperative factors and 0.61 for preoperative factors. Despite lower accuracy under more stringent criteria, the GPT model proved valuable in streamlining the systematic review workflow. Preliminary evaluation of inclusion and exclusion justification provided by the GPT model were reported to have been useful by study screeners, especially in resolving discrepancies during title and abstract screening. Conclusion: This study demonstrates promising use of LLMs to streamline the workflow of systematic reviews. The integration of LLMs in systematic reviews could lead to significant time and cost savings, however caution must be taken for reviews involving stringent a narrower and exclusion criterion. Future research is needed and should explore integrating LLMs in other steps of the systematic review, such as full text screening or data extraction, and compare different LLMs for their effectiveness in various types of systematic reviews."
W7172313156,,10.21203/rs.3.rs-10387804/v1,2026,Mathematical Scientific Discovery Using Large Language Models: A Systematic Literature Review,Vinita Gangaram Jansari,6,Research Square,0,
W4399327517,,10.1101/2024.06.03.24308405,2024,Evaluating the Efficacy of Large Language Models for Systematic Review and Meta-Analysis Screening,Ronald Luo,4,medRxiv,8,"ABSTRACT Background Systematic reviews and meta-analyses are essential for informed research and policymaking, yet they are typically resource-intensive and time-consuming. Recent advances in artificial intelligence and machine learning offer promising opportunities to streamline these processes. Objective To enhance the efficiency of systematic reviews, we explored the automation of various stages using GPT-3.5 Turbo. We assessed the model’s efficacy and performance by comparing it against three expert-conducted reviews across a comprehensive dataset of 24,534 studies. Methods The model’s performance was evaluated through a comparison with three expert reviews, utilizing a pseudo-K-folds permutation and a one-tailed ANOVA with an alpha level of 0.05 to ensure statistical validity. Key performance metrics such as accuracy, sensitivity, specificity, predictive values, F1-score, and the Matthews correlation coefficient were analyzed using two sets of prompts. Results Our approach significantly streamlined the systematic review process, which typically takes a year, reducing it to a few hours without sacrificing quality. In the initial screening phase, accuracy, specificity, and negative predictive values ranged between 80% and 95%. Sensitivity improved markedly during the second screening phase, demonstrating the model’s robustness when provided with more extensive data. Conclusion While ongoing refinements are needed, this tool represents a significant advancement in research methodologies, potentially making systematic reviews more accessible to a wider range of researchers. Impact Statement Our manuscript presents a novel review screening protocol built using open-source frameworks, which significantly enhances the systematic review process in terms of efficiency and cost-effectiveness. Leveraging the capabilities of GPT and embedding models, our protocol demonstrates the potential to transform a traditionally time-consuming and expensive task into an accelerated and economical operation, all while maintaining high standards of accuracy and reliability. Key Points GPT screening can streamline systematic reviews from a year-long, expensive process to just hours at minimal cost. Validated across different topics, the protocol exhibits high reliability and consistency in study inclusion. The AI-driven process reduces human bias, with prompt optimization considerably improving sensitivity."
W7152009027,,10.3390/jcm15082830,2026,Efficacy of Large Language Models for Screening of Systematic Reviews on Periprosthetic Joint Infection,Woojin Shin,6,Journal of Clinical Medicine,0,"Background: Periprosthetic joint infection (PJI) remains a devastating complication following arthroplasty. Systematic reviews of PJI provide essential evidence to inform clinical practice; however, the screening process remains labor-intensive. Recent advancements in large language models (LLMs) offer potential for automating literature screening, though evaluation of current generation models is needed. Methods: This validation study evaluated GPT-5, GPT-5 Pro, and Gemini 2.5 Pro in replicating the title/abstract and full-text screening stages of a published systematic review on intraosseous versus intravenous antibiotic prophylaxis in total joint arthroplasty. Title/abstract screening was performed on 165 articles, followed by a full-text eligibility assessment of 26 articles. Accuracy, sensitivity, specificity, and Cohen’s kappa (κ) were calculated against human screening decisions as the gold standard. Results: In title/abstract screening, GPT-5 Pro achieved the highest accuracy (92.1–92.7%) and specificity (98.6–99.3%), while GPT-5 demonstrated the highest sensitivity (84.6–96.1%). In full-text screening, Gemini 2.5 Pro showed the most consistent performance across repeated evaluations (κ = 0.839 in both trials), whereas GPT-5 Pro exhibited marked intra-model variability (κ = 0.399 to 0.920). Conclusions: Current-generation LLMs achieve near-human accuracy in systematic review screening for PJI research, though substantial intra-model variability underscores the continued need for human oversight in systematic review workflows."
W4405183884,,10.3390/jcm13237480,2024,Exploring the Role of Large Language Models in Melanoma: A Systematic Review,Mor Zarfati,7,Journal of Clinical Medicine,9,"Objective: This systematic review evaluates the current applications, advantages, and challenges of large language models (LLMs) in melanoma care. Methods: A systematic search was conducted in PubMed and Scopus databases for studies published up to 23 July 2024, focusing on the application of LLMs in melanoma. The review adhered to PRISMA guidelines, and the risk of bias was assessed using the modified QUADAS-2 tool. Results: Nine studies were included, categorized into subgroups: patient education, diagnosis, and clinical management. In patient education, LLMs demonstrated high accuracy, though readability often exceeded recommended levels. For diagnosis, multimodal LLMs like GPT-4V showed capabilities in distinguishing melanoma from benign lesions, but accuracy varied, influenced by factors such as image quality and integration of clinical context. Regarding management advice, ChatGPT provided more reliable recommendations compared to other LLMs, but all models lacked depth for individualized decision-making. Conclusions: LLMs, particularly multimodal models, show potential in improving melanoma care. However, current applications require further refinement and validation. Future studies should explore fine-tuning these models on large, diverse dermatological databases and incorporate expert knowledge to address limitations such as generalizability across different populations and skin types."
W4394855484,,10.3390/healthcare12080825,2024,AI and Ethics: A Systematic Review of the Ethical Considerations of Large Language Model Use in Surgery Research,Sophia M. Pressman,6,Healthcare,111,"INTRODUCTION: As large language models receive greater attention in medical research, the investigation of ethical considerations is warranted. This review aims to explore surgery literature to identify ethical concerns surrounding these artificial intelligence models and evaluate how autonomy, beneficence, nonmaleficence, and justice are represented within these ethical discussions to provide insights in order to guide further research and practice. METHODS: A systematic review was conducted in accordance with the Preferred Reporting Items for Systematic Reviews and Meta-Analyses guidelines. Five electronic databases were searched in October 2023. Eligible studies included surgery-related articles that focused on large language models and contained adequate ethical discussion. Study details, including specialty and ethical concerns, were collected. RESULTS: The literature search yielded 1179 articles, with 53 meeting the inclusion criteria. Plastic surgery, orthopedic surgery, and neurosurgery were the most represented surgical specialties. Autonomy was the most explicitly cited ethical principle. The most frequently discussed ethical concern was accuracy (n = 45, 84.9%), followed by bias, patient confidentiality, and responsibility. CONCLUSION: The ethical implications of using large language models in surgery are complex and evolving. The integration of these models into surgery necessitates continuous ethical discourse to ensure responsible and ethical use, balancing technological advancement with human dignity and safety."
W7220675612,,10.21203/rs.3.rs-11150018/v1,2026,An Auditable Large Language Model Pipeline for Data Extraction in Systematic Literature Reviews,Philipp André Macháček,5,Research Square,0,
W4412490346,,10.1038/s41746-025-01824-7,2025,Large language model integrations in cancer decision-making: a systematic review and meta-analysis,Yuexing Hao,7,npj Digital Medicine,66,"Large Language Models (LLMs) are increasingly used to support cancer patients and clinicians in decision-making. This systematic review investigates how LLMs are integrated into oncology and evaluated by researchers. We conducted a comprehensive search across PubMed, Web of Science, Scopus, and the ACM Digital Library through May 2024, identifying 56 studies covering 15 cancer types. The meta-analysis results suggested that LLMs were commonly used to summarize, translate, and communicate clinical information, but performance varied: the average overall accuracy was 76.2%, with average diagnostic accuracy lower at 67.4%, revealing gaps in the clinical readiness of this technology. Most evaluations relied heavily on quantitative datasets and automated methods without human graders, emphasizing ""accuracy"" and ""appropriateness"" while rarely addressing ""safety"", ""harm"", or ""clarity"". Current limitations for LLMs in cancer decision-making, such as limited domain knowledge and dependence on human oversight, demonstrate the need for open datasets and standardized evaluations to improve reliability."
W4391932545,,10.2196/preprints.57400,2024,Large Language Models for Mental Health Applications: Systematic Review (Preprint),Zhijun Guo,6,,24,"BACKGROUND Large language models (LLMs) are advanced artificial neural networks trained on extensive datasets to accurately understand and generate natural language. While they have received much attention and demonstrated potential in digital health, their application in mental health, particularly in clinical settings, has generated considerable debate. OBJECTIVE This systematic review aims to critically assess the use of LLMs in mental health, specifically focusing on their applicability and efficacy in early screening, digital interventions, and clinical settings. By systematically collating and assessing the evidence from current studies, our work analyzes models, methodologies, data sources, and outcomes, thereby highlighting the potential of LLMs in mental health, the challenges they present, and the prospects for their clinical use. METHODS Adhering to the PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) guidelines, this review searched 5 open-access databases: MEDLINE (accessed by PubMed), IEEE Xplore, Scopus, JMIR, and ACM Digital Library. Keywords used were (mental health OR mental illness OR mental disorder OR psychiatry) AND (large language models). This study included articles published between January 1, 2017, and April 30, 2024, and excluded articles published in languages other than English. RESULTS In total, 40 articles were evaluated, including 15 (38%) articles on mental health conditions and suicidal ideation detection through text analysis, 7 (18%) on the use of LLMs as mental health conversational agents, and 18 (45%) on other applications and evaluations of LLMs in mental health. LLMs show good effectiveness in detecting mental health issues and providing accessible, destigmatized eHealth services. However, assessments also indicate that the current risks associated with clinical use might surpass their benefits. These risks include inconsistencies in generated text; the production of hallucinations; and the absence of a comprehensive, benchmarked ethical framework. CONCLUSIONS This systematic review examines the clinical applications of LLMs in mental health, highlighting their potential and inherent risks. The study identifies several issues: the lack of multilingual datasets annotated by experts, concerns regarding the accuracy and reliability of generated content, challenges in interpretability due to the “black box” nature of LLMs, and ongoing ethical dilemmas. These ethical concerns include the absence of a clear, benchmarked ethical framework; data privacy issues; and the potential for overreliance on LLMs by both physicians and patients, which could compromise traditional medical practices. As a result, LLMs should not be considered substitutes for professional mental health services. However, the rapid development of LLMs underscores their potential as valuable clinical aids, emphasizing the need for continued research and development in this area. CLINICALTRIAL PROSPERO CRD42024508617; https://www.crd.york.ac.uk/prospero/display_record.php?RecordID=508617"
W7117820692,,10.3390/app16010368,2025,Retrieval-Augmented Generation (RAG) and Large Language Models (LLMs) for Enterprise Knowledge Management and Document Automation: A Systematic Literature Review,Ehlullah Karakurt,2,Applied Sciences,25,"The integration of Retrieval-Augmented Generation (RAG) with Large Language Models (LLMs) is rapidly transforming enterprise knowledge management, yet a comprehensive understanding of their deployment in real-world workflows remains limited. This study presents a systematic literature review (SLR) analyzing 63 high-quality primary studies selected after rigorous screening to evaluate how these technologies address practical enterprise challenges. We formulated nine research questions targeting platforms, datasets, algorithms, and validation metrics to map the current landscape. Our findings reveal that enterprise adoption is largely in the experimental phase: 63.6% of implementations utilize GPT based models, and 80.5% rely on standard retrieval frameworks such as FAISS or Elasticsearch. Critically, this review identifies a significant ‘lab-to-market’ gap; while retrieval and classification sub-tasks frequently employ academic validation methods like k-fold cross-validation (93.6%), generative evaluation predominantly relies on static hold-out sets due to computational constraints. Furthermore, fewer than 15% of studies address real-time integration challenges required for production scale deployment. By systematically mapping these disparities, this study offers a data-driven perspective and a strategic roadmap for bridging the gap between academic prototypes and robust enterprise applications."
W4388331108,,10.1101/2023.11.04.23298081,2023,Applications of Large Language Models (LLMs) in Breast Cancer Care,Vera Sorin,6,medRxiv,11,"Abstract Purpose Recently introduced Large Language Models (LLMs) such as ChatGPT have already shown promising results in natural language processing in healthcare. The aim of this study is to systematically review the literature on the applications of LLMs in breast cancer diagnosis and care. Methods A literature search was conducted using MEDLINE, focusing on studies published up to October 22nd, 2023, using the following terms: “large language models”, “LLM”, “GPT”, “ChatGPT”, “OpenAI”, and “breast”. Results Five studies met our inclusion criteria. All studies were published in 2023, focusing on ChatGPT-3.5 or GPT-4 by OpenAI. Applications included information extraction from clinical notes, question-answering based on guidelines, and patients’ management recommendations. The rate of correct answers varied from 64-98%, with the highest accuracy (88-98%) observed in information extraction and question-answering tasks. Notably, most studies utilized real patient data rather than data sourced from the internet. Limitations included inconsistent accuracy, prompt sensitivity, and overlooked clinical details, highlighting areas for cautious LLM integration into clinical practice. Conclusion LLMs demonstrate promise in text analysis tasks related to breast cancer care, including information extraction and guideline-based question-answering. However, variations in accuracy and the occurrence of erroneous outputs necessitate validation and oversight. Future works should focus on improving reliability of LLMs within clinical workflow."
W4414365239,,10.1093/ckj/sfaf243,2025,Clinical applications and limitations of large language models in nephrology: a systematic review,Zoe Unger,6,Clinical Kidney Journal,12,"ABSTRACT Background Large language models (LLMs) have emerged as potential tools in healthcare. This systematic review evaluates the applications of text-generative conversational LLMs in nephrology, with particular attention to their reported advantages and limitations. Methods A systematic search was performed in PubMed, Web of Science, Embase and the Cochrane Library in accordance with the Preferred Reporting Items for Systematic Reviews and Meta-Analyses guidelines. Eligible studies assessed LLM applications in nephrology. PROSPERO registration number CRD42024550169. Results Of 1070 records screened, 23 studies met inclusion criteria, addressing four clinical applications in nephrology. In patient education (n = 13), GPT-4 improved the readability of kidney donation information from a 10th to a 4th grade level (9.6 ± 1.9 to 4.30 ± 1.71) and Gemini provided the most accurate answers to chronic kidney disease questions (Global Quality Score 3.46 ± 0.55). Regarding workflow optimization (n = 7), GPT-4 achieved high accuracy (90–94%) in managing continuous renal replacement therapy alarms and improved diagnosis of diabetes insipidus using chain-of-thought and retrieval-augmented prompting. In renal dietary guidance (n = 2), Bard AI led in classifying phosphorus and oxalate content of foods (100% and 84%), while GPT-4 and Bing Chat were most accurate for potassium classification (81%). For laboratory data interpretation (n = 1), Copilot significantly outperformed ChatGPT and Gemini in simulated nephrology datasets (median scores 5/5 compared with 4/5 and 4/5; P < .01). TRIPOD-LLM assessment revealed frequent omissions in data handling, prompting strategies and transparency. Conclusions While LLMs may enhance various aspects of nephrology practice, their widespread adoption remains premature. Input-quality dependence and limited external validation restrict generalizability. Further research is needed to confirm their real-world feasibility and ensure safe clinical integration."
W7156817509,,10.5281/zenodo.19856513,2026,Prompt injection attacks and defenses in large language models: A systematic literature review,Carmine Ambrosino,1,Zenodo (CERN European Organization for Nuclear Research),0,"Prompt injection has rapidly emerged as a critical security threat in the deployment of large language models (LLMs), enabling adversaries to subvert intended behaviors and bypass safety mechanisms. Despite the increased attention that this threat has received, no previous studies have systematically analyzed the field. This paper presents the first systematic literature review (SLR) on prompt injection, with the objective of facilitating a comprehensive, evidence-based understanding of the existing attacks and defenses in LLM among researchers and practitioners.We extensively searched databases like ACM Digital Library, ScienceDirect and Web of Science, initially screening 207 studies and ultimately focusing on 56 relevant papers, based on rigorous inclusion, exclusion, and quality criteria. The analysis is structured around three core research questions: (i) the taxonomical classification of prompt injection attacks, (ii) the identification of recent and innovative attack techniques, and (iii) the characterization of proposed defense mechanisms. The findings reveal a rapidly evolving and multi-layered threat landscape, encompassing obfuscation strategies, automated and multi-modal attacks, and psychologically manipulative prompts. In response, the literature proposes a range of defenses, including input-level sanitization, model-level filtering, prompt engineering, classification-based approaches, and architectural safeguards. Future research should focus on establishing robust standardization in both theory and experimentation, addressing the heterogeneity in attack classification and defense evaluation, while promoting empirical and quantitative approaches to assess effectiveness and considering user privacy and ethical implications."
W7130360482,,10.21203/rs.3.rs-8902070/v1,2026,Large Language Models in Clinical Neurology: A Systematic Review,Alon Gorenshtein,6,Research Square,2,
W4387144848,,10.2196/48996,2023,Automated Paper Screening for Clinical Reviews Using Large Language Models: Data Analysis Study,Eddie Guo,6,Journal of Medical Internet Research,205,"BACKGROUND: The systematic review of clinical research papers is a labor-intensive and time-consuming process that often involves the screening of thousands of titles and abstracts. The accuracy and efficiency of this process are critical for the quality of the review and subsequent health care decisions. Traditional methods rely heavily on human reviewers, often requiring a significant investment of time and resources. OBJECTIVE: This study aims to assess the performance of the OpenAI generative pretrained transformer (GPT) and GPT-4 application programming interfaces (APIs) in accurately and efficiently identifying relevant titles and abstracts from real-world clinical review data sets and comparing their performance against ground truth labeling by 2 independent human reviewers. METHODS: We introduce a novel workflow using the Chat GPT and GPT-4 APIs for screening titles and abstracts in clinical reviews. A Python script was created to make calls to the API with the screening criteria in natural language and a corpus of title and abstract data sets filtered by a minimum of 2 human reviewers. We compared the performance of our model against human-reviewed papers across 6 review papers, screening over 24,000 titles and abstracts. RESULTS: -score of 0.60, a sensitivity of excluded papers of 0.91, and a sensitivity of included papers of 0.76. The interrater variability between 2 independent human screeners was κ=0.46, and the prevalence and bias-adjusted κ between our proposed methods and the consensus-based human decisions was κ=0.96. On a randomly selected subset of papers, the GPT models demonstrated the ability to provide reasoning for their decisions and corrected their initial decisions upon being asked to explain their reasoning for incorrect classifications. CONCLUSIONS: Large language models have the potential to streamline the clinical review process, save valuable time and effort for researchers, and contribute to the overall quality of clinical reviews. By prioritizing the workflow and acting as an aid rather than a replacement for researchers and reviewers, models such as GPT-4 can enhance efficiency and lead to more accurate and reliable conclusions in medical research."
W7133721140,,10.3390/info17030258,2026,"Descriptive Heterogeneity of the Hornblower Sign Across Scientific Literature, Search Engines, and Large Language Models",Peter Melcher,4,Information,0,"Background/Objectives: Digitalization of medical knowledge has improved access to information but also increased the spread of imprecise content. Repeated exposure to incorrect descriptions may lead to their normalization over time. This is particularly evident for the Hornblower sign, which is frequently conflated with the Patte test in the literature, online sources, and large language model outputs. This study systematically evaluates these descriptions and quantifies related inaccuracies. Methods: A three-step approach was applied to answer the question “What is the Hornblower sign?”. First, primary publications referring to the original description by Arthuis were analyzed. Second, the first 35 Google search results were systematically reviewed. Third, responses from five widely used LLMs (ChatGPT 5.1, Grok 4.1, Gemini 3 Pro, Perplexity, and DeepSeek-R1) were evaluated. All descriptions were assessed using a standardized 4-point scoring system (0–3 points) capturing content accuracy and correct differentiation between the Hornblower sign and the Patte test. Results: Fourteen original publications were included, yielding a mean score of 2.07. Correct descriptions were found in 50%, while 43% described only the Patte test. Among 34 evaluable Google search results, the mean score was 1.17, with 77% scoring ≤ 1 point. The five LLMs achieved a mean score of 1.8, demonstrating substantial variability and overall incomplete conceptual accuracy. Conclusions: Descriptions of the Hornblower sign show substantial heterogeneity and frequent inaccuracies across the scientific literature, online sources, and LLM outputs. Conflation with the Patte test undermines diagnostic reliability and limits study comparability. Critical source appraisal and adherence to original test descriptions are essential to maintain clinical and scientific rigor."
W7166039827,,10.1016/j.jval.2026.03.1578,2026,MSR79 AUTOMATION OF SYSTEMATIC REVIEWS WITH LARGE LANGUAGE MODELS,Christian Cao,1,Value in Health,0,
W7167699919,,10.20944/preprints202607.0296.v1,2026,"A Systematic Review of Large Language Models, 2017 to 2026",Prince Samuel Kyeremanteng,3,Preprints.org,0,"Large language models (LLMs) have moved from research artifacts to infrastructure for search, programming, analytics, education, health, public services, and enterprise decision support. The literature has grown quickly, but the evidence is uneven. Some papers report clear gains from scale, instruction tuning, retrieval, and tool use. Others show brittle reasoning, benchmark leakage, privacy exposure, harmful outputs, and weak documentation. This systematic review maps the technical and governance evidence on LLMs from 2017 to 2026 using a PRISMA 2020 guided process. Searches were conducted in OpenAlex and Crossref on June 5, 2026, with citation chasing for seminal model, benchmark, alignment, retrieval, agent, and risk papers. The final synthesis includes 85 validated studies, each checked through DOI, arXiv, ACL Anthology, ACM, OpenReview, publisher, or official technical-report metadata. Results are organized around ten themes: architecture and scaling, data and documentation, alignment and adaptation, prompting and reasoning, retrieval and grounding, tools and agents, evaluation, risk and governance, domain-specialized models, and multimodal extensions. The review finds that LLM progress is not explained by parameter count alone. Capability depends on data composition, compute allocation, post-training, interfaces, retrieval design, evaluation protocol, and deployment controls. For industry, the most defensible path is not blind adoption or blanket rejection. It is task specific evaluation, provenance-aware retrieval, staged release, documented failure handling, privacy testing, human accountability, and continuous monitoring. The paper closes with a research agenda for evidence that is more reproducible, deployment-aware, and useful outside leaderboards."
W4414834072,10.48550/arxiv.2502.15702,10.48550/arxiv.2502.15702,2025,Large language models streamline automated systematic review: A preliminary study,Xi Chen,2,arXiv (Cornell University),2,"Large Language Models (LLMs) have shown promise in natural language processing tasks, with the potential to automate systematic reviews. This study evaluates the performance of three state-of-the-art LLMs in conducting systematic review tasks. We assessed GPT-4, Claude-3, and Mistral 8x7B across four systematic review tasks: study design formulation, search strategy development, literature screening, and data extraction. Sourced from a previously published systematic review, we provided reference standard including standard PICO (Population, Intervention, Comparison, Outcome) design, standard eligibility criteria, and data from 20 reference literature. Three investigators evaluated the quality of study design and eligibility criteria using 5-point Liker Scale in terms of accuracy, integrity, relevance, consistency and overall performance. For other tasks, the output is defined as accurate if it is the same as the reference standard. Search strategy performance was evaluated through accuracy and retrieval efficacy. Screening accuracy was assessed for both abstracts screening and full texts screening. Data extraction accuracy was evaluated across 1,120 data points comprising 3,360 individual fields. Claude-3 demonstrated superior overall performance in PICO design. In search strategy formulation, GPT-4 and Claude-3 achieved comparable accuracy, outperforming Mistral. For abstract screening, GPT-4 achieved the highest accuracy, followed by Mistral and Claude-3. In data extraction, GPT-4 significantly outperformed other models. LLMs demonstrate potential for automating systematic review tasks, with GPT-4 showing superior performance in search strategy formulation, literature screening and data extraction. These capabilities make them promising assistive tools for researchers and warrant further development and validation in this field."
W4416589125,,10.1016/j.ijmedinf.2025.106205,2025,AI-assisted literature screening: A hybrid approach using large language models and retrieval-augmented generation,Yiming Li,10,International Journal of Medical Informatics,8,
W7117516269,,10.1109/cinti67731.2025.11311831,2025,Large Language Models for Title/Abstract Screening in Systematic Literature Reviews: A Case Study in Precision Livestock Farming,Márk Zrubka,4,IEEE International Symposium on Computational Intelligence and Informatics (CINTI),2,"Large language models (LLMs) are gaining popularity in systematic literature review automation, and they are even acknowledged as second reviewers along humans by the Cochrane Handbook. In this study, we tested the performance of two LLMs in the title/abstract screening process in a systematic review on precision livestock farming costbenefit analyses. We assessed the screening accuracy, sensitivity and specificity of the GPT-4o Mini and Claude 3.5 Sonnet models, as well as a human rater. We also compared the time and costs of completing the title/abstract screening by two human raters versus one human assisted by LLMs. LLMassisted screening required less time and money than two human raters, but it was also less accurate, resulting in extra work to rescreen discrepancies. Nevertheless, combining LLMs with the human ratings improved the quality of the screening output. Our results suggest that the tested LLMs are not as reliable as expert humans, however, they can assist in the title/abstract screening process and improve the quality of work with very low extra resources."
W7203580816,,10.5281/zenodo.21967539,2026,Large Language Models as Research Assistants: A Cross-Disciplinary Systematic Literature Review,Indian Institute of Technology Bombay,1,Zenodo (CERN European Organization for Nuclear Research),0,"Large language models have been absorbed into research practice across nearly every discipline in under four years, faster than the evidence base needed to evaluate them could be assembled. This review synthesises 124 sources published between January 2020 and June 2026, identified through structured searches of seven databases, and organises the evidence by research lifecycle stage rather than by model architecture. Randomised evidence shows substantial gains on bounded, verifiable, language-intensive work - a forty per cent reduction in task time in one experiment, twelve per cent more tasks completed at twenty-five per cent greater speed in another - with effects concentrated among initially lower-performing users, alongside sharp degradation on tasks just beyond the model's competence. Reliability failures are structural rather than incidental: fabricated bibliographic references fell from roughly fifty-five per cent in GPT-3.5 to eighteen per cent in GPT-4, an improvement far short of what unverified citation practice requires, while corpus analysis indicates at least 13.5 per cent of 2024 biomedical abstracts show signs of machine assistance. Governance has converged on three norms - no machine authorship, mandatory disclosure, and exclusion from confidential peer review - while leaving verification, provenance and researcher training largely unaddressed. The review proposes a five-level scheme of research autonomy and a verification protocol calibrated to the cost of undetected error, and argues that the field's central deficit is evaluative: benchmarks measure question answering, not whether assisted research is better research."
W7171006779,,,2026,MechaScreener: Large Language Model-Based Automated Screening for Systematic Reviews and Research,Carly Hudson,5,OSF Preprints (OSF Preprints),0,"Systematic Reviews (SRs) are the gold standard for evidence synthesis, but the manual title and abstract screening of thousands of references creates a severe bottleneck. Existing automated tools have historically struggled to achieve the near-perfect recall (sensitivity) required for reliable reviews. We developed MechaScreener as a ""zero-shot"" automated screening tool that utilises a Large Language Model (LLM) to rank article relevance. The tool requires no initial training data or manual pre-screening, as MechaScreener directly applies user-provided question elements (PICO) or inclusion/exclusion criteria to assign an ordinal relevance score (1-5) to each reference. We evaluated the tool in two phases: a development phase using five reference libraries to optimise prompts, and an independent evaluation phase using 10 diverse Cochrane review libraries (comprising both randomised controlled trials and non-RCTs) containing over 58,000 references. In the evaluation dataset, MechaScreener achieved a perfect recall of 1.00 (100\%) in all 10 reviews, ensuring no relevant articles were missed (pooled record-level 95% CI: 0.98-1.00). Concurrently, it achieved an overall macro-average specificity of 0.61 (61%, cluster bootstrap 95% CI: 0.46-0.75; pooled record-level specificity 0.59, 95% CI: 0.59-0.60). Specificity varied: from 0.21 in broad public health topics to 0.91 in precise pharmacological interventions; reflecting the tool's built-in conservatism when evaluating ambiguous abstracts. By safely eliminating over 60\% of irrelevant literature during the initial screening phase without compromising recall, MechaScreener functions as a highly reliable but low-effort ``first-pass'' filter, allowing researchers to substantially reduce manual workloads and reallocate resources toward full-text review and data extraction."
W7170048440,,10.17605/osf.io/u43m2,2026,MechaScreener: Large Language Model-Based Automated Screening for Systematic Reviews and Research,Connor Forbes,5,Open Science Framework,0,"Systematic Reviews (SRs) are the gold standard for evidence synthesis, but the manual title and abstract screening of thousands of references creates a severe bottleneck. Existing automated tools have historically struggled to achieve the near-perfect recall (sensitivity) required for reliable reviews. We developed MechaScreener as a ""zero-shot"" automated screening tool that utilises a Large Language Model (LLM) to rank article relevance. The tool requires no initial training data or manual pre-screening, as MechaScreener directly applies user-provided question elements (PICO) or inclusion/exclusion criteria to assign an ordinal relevance score (1-5) to each reference. We evaluated the tool in two phases: a development phase using five reference libraries to optimise prompts, and an independent evaluation phase using 10 diverse Cochrane review libraries (comprising both randomised controlled trials and non-RCTs) containing over 58,000 references. In the evaluation dataset, MechaScreener achieved a perfect recall of 1.00 (100\%) in all 10 reviews, ensuring no relevant articles were missed (pooled record-level 95% CI: 0.98-1.00). Concurrently, it achieved an overall macro-average specificity of 0.61 (61%, cluster bootstrap 95% CI: 0.46-0.75; pooled record-level specificity 0.59, 95% CI: 0.59-0.60). Specificity varied: from 0.21 in broad public health topics to 0.91 in precise pharmacological interventions; reflecting the tool's built-in conservatism when evaluating ambiguous abstracts. By safely eliminating over 60\% of irrelevant literature during the initial screening phase without compromising recall, MechaScreener functions as a highly reliable but low-effort ``first-pass'' filter, allowing researchers to substantially reduce manual workloads and reallocate resources toward full-text review and data extraction."
W7161880413,,10.3389/or.2026.1757059,2026,Applications of large language models in tumor boards: a systematic review,Benjamin Konzmann,4,Oncology Reviews,4,"Background: Multidisciplinary tumor boards (MDTs) are the gold standard for cancer care but currently face significant pressure from rising case volumes and the increasing complexity of precision medicine. Large Language Models (LLMs) offer potential as Clinical Decision Support Systems (CDSS) to augment these workflows. This systematic review evaluates the current applications, accuracy, and safety of LLMs in MDT decision-making. Methods: A systematic review was conducted following PRISMA 2020 guidelines. We searched PubMed/MEDLINE, Embase, IEEE Xplore, and arXiv for primary research published between January 2023 and November 2025. Inclusion criteria required studies to evaluate LLM performance in tumor board settings against human consensus or established clinical guidelines. Results: Thirty-one studies encompassing 3,845 unique patient cases were included. The analysis revealed a distinct ""complexity gap"" in model performance. In standardized, high-incidence domains such as breast and prostate cancer, advanced models (particularly the GPT-4 family) demonstrated high concordance (up to 94%) with human experts. However, performance degraded significantly in complex, rare, or multimodal scenarios-such as sarcoma or neuro-oncology-where models struggled with ""gray zone"" decision-making and lacked the ability to interpret non-textual data. While LLMs showed utility in administrative tasks and guideline retrieval, they remained prone to hallucinations and lacked the nuance required for holistic patient assessment. Conclusion: Current LLMs exhibit sufficient maturity to function as assistive tools for documentation and decision support in routine oncological cases but are not yet reliable enough for autonomous decision-making. Successful clinical implementation will require ""human-in-the-loop"" safeguards, the development of multimodal architectures, and rigorous prospective validation to ensure patient safety."
W4416534882,,10.1016/j.infsof.2026.108204,2026,LLM4SCREENLIT: Recommendations on assessing the performance of large language models for screening literature in systematic reviews,Lech Madeyski,3,Information and Software Technology,4,"Context: Large language models (LLMs) are increasingly used to screen literature for systematic reviews (SRs), but the standard confusion-matrix metrics used to evaluate them can mislead under the imbalanced, cost-asymmetric conditions of screening. Objective: We develop and justify LLM4SCREENLIT—practical recommendations for researchers conducting LLM-screening evaluations and for editors and reviewers assessing such studies—differentiated by study type (retrospective benchmarking vs. deployment for a specific SR). Method: Using Delgado-Chaves et al. (2025), an 18-LLM benchmark across three biomedical SRs, as a motivating example, we reviewed 28 additional papers and extracted their reported metrics. We propose a Weighted Matthews Correlation Coefficient (WMCC) that integrates MCC’s chance-correction with asymmetric misclassification costs, and validated it on three software-engineering (SE) reanalyses (Felizardo et al. 2024; Syriani et al. 2024; Huotala et al. 2025), the largest covering 9 LLMs × 24 SE secondary studies (34,528 articles). Results: Across the 29 papers, only 10% reported MCC, only 24% reported full confusion matrices, and none of the five papers claiming workload savings priced false-negative cost. In the largest SE reanalysis, MCC and WMCC disagree on the best LLM in 55% of evaluable studies; in the most striking 9695-article SE study, the Accuracy-best LLM loses 63.3% of relevant evidence (Lost Evidence), the MCC-best 43.9%, but the WMCC-best only 5.8%. Sensitivity analysis (median crossover at w ≈ 2 . 7 , all < 7 ) supports w = 10 as a conservative default. Conclusions: SR-screening evaluations should prioritize Lost Evidence and use cost-sensitive WMCC alongside MCC for ranking. Reporting must include the full confusion matrix and treat unclassifiable outputs as positives requiring human review. Designs should be leakage-aware, with non-LLM baselines when the study aims to inform SR practice and labels are available. Editors and reviewers should require these elements as routine. Extension to full-text screening and data extraction is principled but pending empirical validation."
W4410564944,,10.3390/info16050420,2025,PROMPTHEUS: A Human-Centered Pipeline to Streamline Systematic Literature Reviews with Large Language Models,Joao Torres,4,Information,8,"The growing volume of academic publications poses significant challenges for researchers conducting timely and accurate systematic literature reviews (SLR), particularly in fast-evolving fields like artificial intelligence. This growth of academic literature also makes it increasingly difficult for lay people to access scientific knowledge effectively, meaning academic literature is often misrepresented in the popular press and, more broadly, in society. Traditional SLRs are labor-intensive and error-prone, and they struggle to keep up with the rapid pace of new research. To address these issues, we developed PROMPTHEUS: an AI-driven pipeline solution that automates the systematic literature review (LR) process using large language models (LLMs). We aimed to enhance efficiency by reducing the manual workload while maintaining the precision and coherence required for comprehensive literature synthesis. PROMPTHEUS automates key SLR stages, including systematic searches, data extraction, and topic modeling using BERTopic and summarization with transformer models. Evaluations across five research domains demonstrated that PROMPTHEUS reduces review time, achieves high precision, and provides coherent topic organization, offering a scalable and effective solution for conducting literature reviews in an increasingly crowded research landscape."
W7202271496,,10.1002/lary.70801,2026,ENTGPT : Applying Large Language Models to Systematic Review Screening With the Novel STARR Protocol,Akash Kapoor,6,The Laryngoscope,0,"OBJECTIVES: Systematic literature reviews (SLRs) are time-intensive and resource-consuming. While large language models (LLMs) have shown promise in encoding clinical knowledge, evidence for their performance on complex text analysis, necessary for SLRs in otolaryngology, remains limited. In this proof-of-concept study, we investigate an LLM's performance in screening relevant articles for a SLR with the novel screening of title and abstracts, reevaluation, and full-text review (STARR) protocol. METHODS: ENTGPT (based on GPT-4o) was compared to two human reviewers in article inclusion/exclusion decisions using the traditional and STARR screening protocols. ENTGPT was provided with inclusion and exclusion criteria, titles, abstracts, and full texts (if available) of the 850 articles retrieved in the original search. The model's decisions were compared to those made by two human reviewers. RESULTS: ENTGPT, using the STARR protocol, achieved 99.87% accuracy in article classification compared with human reviewers (95% CI: 0.99-1.0), including 100% specificity and 95% sensitivity. When using the traditional protocol, sensitivity declined to 35%. ENTGPT, using the traditional protocol, achieved 99.47% accuracy in article classification compared with human reviewers (95% CI: 0.99-1.0), including 100% specificity and 35% sensitivity. When using the STARR protocol, accuracy improved to 99.87% and sensitivity markedly increased to 95%. CONCLUSIONS: ENTGPT accurately replicated human reviewers in article selection and data extraction for an otolaryngology SLR using the STARR and traditional protocols. This performance suggests that LLMs could be employed to significantly streamline the SLR process, potentially saving substantial time and resources for researchers. LEVEL OF EVIDENCE: N/A."
W7166589016,,10.63317/5dnbkz4d44kz,2026,Automatic Generation of Discharge Summaries Using Large Language Models: A Systematic Literature Review,Lucas Molino-Piñar,3,Language Resources and Evaluation Conference (LREC),0,"Discharge summaries are critical documents for continuity of care, yet their manual creation imposes significant burdens on clinical staff.This systematic literature review examines current approaches to automatic generation of discharge summaries using Natural Language Processing (NLP) and Large Language Models (LLMs).Following the Kitchenham guidelines for systematic reviews in software engineering, we searched Scopus and PubMed databases for studies published between 2023 and 2026, identifying 9 primary studies from an initial pool of 102 papers.Our analysis reveals that GPT-4 and its variants dominate current research (appearing in 6 of 9 studies), while open-source alternatives like LLaMA show promise for privacy-preserving deployments.Evaluation primarily relies on automatic metrics (ROUGE, BLEU) combined with human expert assessment.Key challenges include hallucination rates ranging from 33% to 64%, information omission, integration with Electronic Health Record (EHR) systems, and context window limitations.Studies addressing factuality employ human-in-the-loop validation, prompt engineering techniques, and knowledge graph-based correction mechanisms.Despite these challenges, recent implementations demonstrate clinical feasibility, with one study achieving a 94.35% System Usability Score.This review provides a comprehensive synthesis of the state-of-the-art and identifies opportunities for future research in this rapidly evolving field."
W7160102024,,10.1017/rsm.2026.10124,2026,MechaScreener: Large language model-based automated screening for systematic reviews and research,Connor Forbes,5,Research Synthesis Methods,0,"Systematic reviews are the gold standard for evidence synthesis, but the manual title and abstract screening of thousands of references creates a severe bottleneck. Existing automated tools have historically struggled to achieve the near-perfect recall (sensitivity) required for reliable reviews. We developed MechaScreener as a 'zero-shot' automated screening tool that utilises a large language model to rank article relevance. The tool requires no initial training data or manual pre-screening, as MechaScreener directly applies user-provided question elements (PICO) or inclusion/exclusion criteria to assign an ordinal relevance score (1-5) to each reference. We evaluated the tool in two phases: a development phase using five reference libraries to optimise prompts, and an independent evaluation phase using 10 diverse Cochrane review libraries (comprising both randomised controlled trials [RCTs] and non-RCTs) containing over 58,000 references. In the evaluation dataset, MechaScreener achieved a perfect recall of 1.00 (100%) in all 10 reviews, ensuring that no relevant articles were missed (pooled record-level 95% CI: 0.98-1.00). Concurrently, it achieved an overall macro-average specificity of 0.61 (61%, cluster bootstrap 95% CI: 0.46-0.75; pooled record-level specificity 0.59, 95% CI: 0.59-0.60). Specificity varied from 0.21 in broad public health topics to 0.91 in precise pharmacological interventions, reflecting the tool's built-in conservatism when evaluating ambiguous abstracts. By safely eliminating over 60% of irrelevant literature during the initial screening phase without compromising recall, MechaScreener functions as a highly reliable but low-effort 'first-pass' filter, allowing researchers to substantially reduce manual workloads and reallocate resources towards full-text review and data extraction."
W7140887005,,10.1145/3802827,2026,Enhancing Automated Unit Test Generation with Large Language Models: A Systematic Literature Review,Junwei Zhang,5,ACM Transactions on Software Engineering and Methodology,4,"Automated unit test generation is a fundamental yet challenging task in software engineering (SE), playing a critical role in ensuring software correctness, reliability, and maintainability. While traditional approaches such as search-based software testing and symbolic execution have achieved notable success, they often suffer from limited semantic understanding, high configuration costs, and scalability constraints. Recent advances in Large Language Models (LLMs) have fundamentally reshaped the landscape of automated unit testing by enabling models to reason over source code semantics and generate executable, context-aware test cases. Despite the rapid growth of this research area, a comprehensive and task-oriented synthesis of existing work remains lacking. This article presents a systematic literature review of LLM-based unit test generation. This review draws on research from leading SE and AI conferences and journals, including 69 papers published across 25 distinct venues, along with 47 high-quality preprint papers, bringing the total to 116. Our review aims to answer three key research questions: (1) which unit testing tasks have been addressed using LLMs, (2) how LLMs are adapted and integrated into the unit test generation pipeline, and (3) what datasets, benchmarks, and evaluation practices are employed in existing studies. To this end, we organize the literature from a task-centric perspective, covering test generation, test input generation, test oracle generation, and test evolution, and from a methodological perspective, categorizing LLM adaptation strategies into fine-tuning, prompt engineering, and agent-based approaches. Our analysis reveals that current research predominantly focuses on function- and class-level test generation, with comparatively limited attention given to test input generation, oracle construction, and long-term test evolution. Decoder-only LLMs, particularly GPT-family and LLaMA-based models, dominate the field, while encoder-only and encoder–decoder models remain underexplored. We further observe substantial disparities in dataset characteristics, programming language coverage, and evaluation metrics, which hinder fair comparison and reproducibility across studies. Based on empirical evidence extracted from the surveyed literature, we identify key challenges facing LLM-based unit test generation. Building on these findings, we outline several promising research directions, such as dataset optimization, structure-aware context modeling, agent coordination mechanisms, and benchmark enhancement. This review provides a consolidated and evidence-driven foundation for future research, aiming to advance the development of scalable, reliable, and practically applicable LLM-driven unit testing techniques."
W7147185311,,10.1109/iccsm66818.2025.00027,2025,Toward the Conduction of Automated Systematic Literature Reviews: The Role of Large Language Models,Marco Fiore,1,"International Conference on Computer, Software and Modeling (ICCSM)",0,"The conduction of Systematic Literature Reviews is an essential yet labor-intensive process for researchers, requiring meticulous compilation and analysis of literature to answer predefined Research Questions. Despite attempts to streamline this process, automation of reviews remains a significant challenge. This paper explores the potential of Generative AI and Large Language Models to revolutionize the review process by automating key tasks such as identifying, extracting, and synthesizing relevant literature. By leveraging open-source Large Language Models, we aim to reduce the time and effort traditionally required for comprehensive literature reviews. Our study involves comparing various models, including lightweight and more robust ones. We replicate a published review on Blockchain technology in agri-food traceability to evaluate the performance of these models. Results indicate that while high-parameter models successfully generate relevant search strings and classifications, they also exhibit trade-offs in terms of response times. Our findings highlight both the promise and limitations of current Large Language Models in automating Systematic Literature Reviews."
W7155247769,,,2026,Large Language Models in Factory Planning: A Systematic Literature Review of an Emerging Research Field,Richard Reider,1,OSF Preprints (OSF Preprints),0,"This project is a systematic literature review of large language models (LLMs) and agentic AI systems applied to factory planning as defined by VDI 5200 and the framework of Wiendahl, Reichardt & Nyhuis — covering strategic, concept, and detail planning phases from requirements elicitation to layout design. Motivated by the rapid emergence of LLM-based assistance systems in manufacturing since 2023 and the absence of a dedicated synthesis at the factory-planning level, the review asks what tasks LLMs currently address, which architectural patterns (RAG, multi-agent systems, LLM+solver/digital-twin hybrids) have emerged, what evidence exists on effectiveness, and where the research gaps lie. The search follows PRISMA 2020 and combines a database track (Scopus TITLE-ABS-KEY, Web of Science TS=, and OpenAlex full-text) with a parallel hand-search track covering German grey literature (Fraunhofer Publica, ZWF archive, mediaTUM, KLUEDO) and seed-paper snowballing from anchor works by Ferry/Huang/Reinhart (ZWF 2025), Klar et al. (JMS 2024), and the Fraunhofer GenAI4FFD project. Screening uses dual independent review on a 10% sample with Cohen's κ ≥ 0.7 before single-rater continuation, and a tiered quality rubric weights venue rank, methodological transparency, and empirical grounding during synthesis. All materials — protocol, search strings, screening logs, extraction tables, and PRISMA flow diagram — are shared openly to ensure reproducibility and to support living-review updates as the field matures."
W4415271247,,10.1007/s10462-025-11398-1,2025,A systematic review of multi-modal large language models on domain-specific applications,Sirui Li,4,Artificial Intelligence Review,19,"Abstract While Large Language Models (LLMs) have shown remarkable proficiency in text-based tasks, they struggle to interact effectively with the more realistic world without the perceptions of other modalities such as visual and audio. Multi-modal LLMs, which integrate these additional modalities, have become increasingly important across various domains. Despite the significant advancements and potential of multi-modal LLMs, there has been no comprehensive PRISMA-based systematic review that examines their applications across different domains. The objective of this work is to fill this gap by systematically reviewing and synthesising the quantitative research literature on domain-specific applications of multi-modal LLMs. This systematic review follows the PRISMA guidelines to analyse research literature published after 2022, the release of OpenAI’s ChatGPT $$-$$ 3.5. The literature search was conducted across several online databases, including Nature, Scopus, and Google Scholar. A total of 22 studies were identified, with 11 focusing on the medical domain, 3 on autonomous driving, and 2 on geometric analysis. The remaining studies covered a range of topics, with one each on climate, music, e-commerce, sentiment analysis, human-robot interaction, and construction. This review provides a comprehensive overview of the current state of multi-modal LLMs, highlights their domain-specific applications, and identifies gaps and future research directions."
W4401633394,,10.2196/52758,2024,Human-Comparable Sensitivity of Large Language Models in Identifying Eligible Studies Through Title and Abstract Screening: 3-Layer Strategy Using GPT-3.5 and GPT-4 for Systematic Reviews,Kentaro Matsui,6,Journal of Medical Internet Research,50,"BACKGROUND: The screening process for systematic reviews is resource-intensive. Although previous machine learning solutions have reported reductions in workload, they risked excluding relevant papers. OBJECTIVE: We evaluated the performance of a 3-layer screening method using GPT-3.5 and GPT-4 to streamline the title and abstract-screening process for systematic reviews. Our goal is to develop a screening method that maximizes sensitivity for identifying relevant records. METHODS: We conducted screenings on 2 of our previous systematic reviews related to the treatment of bipolar disorder, with 1381 records from the first review and 3146 from the second. Screenings were conducted using GPT-3.5 (gpt-3.5-turbo-0125) and GPT-4 (gpt-4-0125-preview) across three layers: (1) research design, (2) target patients, and (3) interventions and controls. The 3-layer screening was conducted using prompts tailored to each study. During this process, information extraction according to each study's inclusion criteria and optimization for screening were carried out using a GPT-4-based flow without manual adjustments. Records were evaluated at each layer, and those meeting the inclusion criteria at all layers were subsequently judged as included. RESULTS: On each layer, both GPT-3.5 and GPT-4 were able to process about 110 records per minute, and the total time required for screening the first and second studies was approximately 1 hour and 2 hours, respectively. In the first study, the sensitivities/specificities of the GPT-3.5 and GPT-4 were 0.900/0.709 and 0.806/0.996, respectively. Both screenings by GPT-3.5 and GPT-4 judged all 6 records used for the meta-analysis as included. In the second study, the sensitivities/specificities of the GPT-3.5 and GPT-4 were 0.958/0.116 and 0.875/0.855, respectively. The sensitivities for the relevant records align with those of human evaluators: 0.867-1.000 for the first study and 0.776-0.979 for the second study. Both screenings by GPT-3.5 and GPT-4 judged all 9 records used for the meta-analysis as included. After accounting for justifiably excluded records by GPT-4, the sensitivities/specificities of the GPT-4 screening were 0.962/0.996 in the first study and 0.943/0.855 in the second study. Further investigation indicated that the cases incorrectly excluded by GPT-3.5 were due to a lack of domain knowledge, while the cases incorrectly excluded by GPT-4 were due to misinterpretations of the inclusion criteria. CONCLUSIONS: Our 3-layer screening method with GPT-4 demonstrated acceptable level of sensitivity and specificity that supports its practical application in systematic review screenings. Future research should aim to generalize this approach and explore its effectiveness in diverse settings, both medical and nonmedical, to fully establish its use and operational feasibility."
W4415949686,,10.1017/rsm.2026.10110,2026,Evaluating prompting strategies and large language models in systematic literature review screening: Relevance and task-stage classification,Binglan Han,4,Research Synthesis Methods,0,"This study quantifies how prompting strategies interact with large language models (LLMs) to automate the screening stage of systematic literature reviews (SLRs). We evaluate six LLMs (GPT-4o, GPT-4o-mini, DeepSeek-Chat-V3, Gemini-2.5-Flash, Claude-3.5-Haiku, and Llama-4-Maverick) under five prompt types (zero-shot, few-shot, chain-of-thought [CoT], CoT-few-shot, and self-reflection) across relevance classification and six Level-2 tasks, using accuracy, precision, recall, and F1. Results show pronounced model-prompt interaction effects within this SLR-automation corpus: zero-shot maximizes recall for high-sensitivity relevance screening; CoT-few-shot yields stronger precision and F1; and self-reflection underperforms due to over-inclusivity and instability across models. GPT-4o and DeepSeek provide robust overall performance, while GPT-4o-mini performs competitively at a substantially lower dollar cost. A cost-performance analysis for relevance classification (per 1,000 abstracts) reveals large absolute differences among model-prompt pairings; GPT-4o-Mini remains low-cost across prompts, and structured prompts (CoT/CoT-few-shot) on GPT-4o-Mini offer attractive F1 at a small incremental cost. The cost-performance analysis also suggests that future screening workflows could incorporate staged, cost-aware procedures: (1) using lower-cost, high-recall configurations for initial screening and (2) reserving higher-cost or more precision-oriented configurations for later triage or uncertain records. These findings highlight LLMs' uneven but promising potential to support screening within corpora similar to the SLR-automation corpus examined in this study. Further validation is needed before generalising these results to broader systematic review screening tasks. By systematically analyzing prompt-model interactions, we provide a comparative benchmark and practical guidance for task-adaptive LLM deployment."
W4417081257,,10.1007/s10758-025-09922-x,2025,Large Language Model Applications in the Algebra Domain: A Systematic Review,Yajie Song,5,Technology Knowledge and Learning,0,
W4409794823,,10.2196/64963,2025,Comparing Diagnostic Accuracy of Clinical Professionals and Large Language Models: Systematic Review and Meta-Analysis,Guxue Shan,7,JMIR Medical Informatics,71,"Background: With the rapid development of artificial intelligence (AI) technology, especially generative AI, large language models (LLMs) have shown great potential in the medical field. Through massive medical data training, it can understand complex medical texts and can quickly analyze medical records and provide health counseling and diagnostic advice directly, especially in rare diseases. However, no study has yet compared and extensively discussed the diagnostic performance of LLMs with that of physicians. Objective: This study systematically reviewed the accuracy of LLMs in clinical diagnosis and provided reference for further clinical application. Methods: We conducted searches in CNKI (China National Knowledge Infrastructure), VIP Database, SinoMed, PubMed, Web of Science, Embase, and CINAHL (Cumulative Index to Nursing and Allied Health Literature) from January 1, 2017, to the present. A total of 2 reviewers independently screened the literature and extracted relevant information. The risk of bias was assessed using the Prediction Model Risk of Bias Assessment Tool (PROBAST), which evaluates both the risk of bias and the applicability of included studies. Results: A total of 30 studies involving 19 LLMs and a total of 4762 cases were included. The quality assessment indicated a high risk of bias in the majority of studies, primary cause is known case diagnosis. For the optimal model, the accuracy of the primary diagnosis ranged from 25% to 97.8%, while the triage accuracy ranged from 66.5% to 98%. Conclusions: LLMs have demonstrated considerable diagnostic capabilities and significant potential for application across various clinical cases. Although their accuracy still falls short of that of clinical professionals, if used cautiously, they have the potential to become one of the best intelligent assistants in the field of human health care."
W7116974613,,10.2196/preprints.89862,2025,Large Language Models in Colorectal Cancer: A Systematic Review (Preprint),Jinglei Tian,6,,0,"UNSTRUCTURED This systematic review aimed to analyze the application and efficacy of large language models (LLMs) in the full-cycle management of colorectal cancer (CRC). CRC diagnosis and treatment face challenges in multi-stage collaboration and integration of large-scale clinical data. While LLMs offer potential solutions, targeted evidence in CRC-specific settings remains insufficient. Following PRISMA guidelines, six databases including PubMed, Embase, and Web of Science were systematically searched up to November 1, 2025. Two reviewers independently screened studies, extracted data, and assessed quality using appropriate tools (QUADAS-2, PROBAST, ROBINS-I) based on study design. A total of 34 studies were included. LLMs were primarily applied in auxiliary diagnosis, information extraction, knowledge-based question answering, treatment decision support, and predictive modeling. GPT-4 achieved a diagnostic accuracy of 97.2% and an F1-score of 84.1% in information extraction, while lightweight or domain-specialized models (e.g., Gemma-2, RoBERTa) performed optimally in specific tasks. Significant heterogeneity was observed across studies due to variations in model types, prompt engineering strategies, and outcome measures. In conclusion, LLMs demonstrate considerable potential in supporting CRC management, particularly in data structuring and decision-making. However, limitations include data bias, model hallucination, and insufficient generalizability. Future research should focus on multicenter prospective validation, improvement of model interpretability, and development of multimodal and domain-adapted LLMs for safe and effective clinical integration."
W4408680354,,10.2196/preprints.74177,2025,Large Language Models in Lung Cancer: A Systematic Review (Preprint),Ruikang Zhong,9,,0,"BACKGROUND In the era of data and intelligence, Artificial Intelligence (AI), especially Large Language Model (LLM), has been widely applied in the medical field. OBJECTIVE The aim of this systematic review is to discuss the progress of LLM application in lung cancer, including clinical, educational and research aspects, and to explore the potential of LLM application in the full-cycle management of lung cancer. METHODS We searched six electronic databases according to PRISMA guidelines. Information was screened and extracted independently by two authors according to the inclusion criteria. Quality assessment was performed using the QUADAS-2, PROBAST and ROBINS-I tools. RESULTS The literature search yielded 706 relevant studies, resulting in the inclusion of a total of 28 studies published between 2023 and 2024.LLMs are widely used in lung cancer-related tasks, including assisted diagnosis, information extraction, question and answer science, and therapeutic decision support.ChatGPT is the most commonly used model and shows significant potential for improving diagnostic accuracy and patient communication. The importance of cue engineering and fine-tuning in optimising the performance of LLMs in specific clinical tasks is also highlighted. CONCLUSIONS LLMs have the potential to transform lung cancer management by improving diagnostic accuracy, patient communication and treatment planning. However, issues such as data security, ethical regulations, economic costs and clinical validation need to be addressed and LLMs need to be used rationally as effective tools rather than misused or even replaced by healthcare professionals."
W7135056671,,10.1109/sami68106.2026.11420686,2026,Assessing Probability Rating Performance for Large Language Models in Systematic Literature Review Automation,Márk Zrubka,3,IEEE World Symposium on Applied Machine Intelligence and Informatics (SAMI),1,"Classical machine learning methods of systematic literature review (SLR) automation use thresholded probability scores to make decisions about including or excluding articles in the title/abstract selection process. Large language models (LLMs) have gained popularity in SLR automation and are sometimes prompted to give similar probability score outputs as classical models. LLMs do not always perform well in mathematical reasoning tasks, which raises the question whether their probability outputs are meaningful. We evaluated the accuracy of four LLMs in title/abstract screening, using qualitative rating (Yes/Unsure/No) and a continuous probability rating (0-1). We also assessed the consistency between probability and qualitative ratings within each model using violin plots. The results show that LLMs performed well in excluding irrelevant articles, but their performance dropped notably for including relevant articles. Claude Sonnet 4 and Phi 4 produced consistent probability and qualitative ratings. GPT 4o-mini did not represent probabilities correctly and Qwen 2.5 underperformed in the qualitative category rating. Due to the varying performance of LLMs, we recommend using multiple LLMs to assist human reviewers in the title/abstract screening process of SLRs. Furthermore, we advise testing whether the output of LLMs is meaningful before using them to evaluate articles."
W4414794728,,10.3233/shti251488,2025,Leveraging ChatGPT-4 for Evidence Synthesis: A Case Study on the Use of a Large Language Model in a Systematic Review,Federica Tomassini,6,Studies in health technology and informatics,0,"Artificial intelligence, particularly Large Language Models (LLM) such as ChatGPT, is emerging as a potentially transformative support for traditionally complex and time-consuming Systematic Literature Reviews (SLRs). In this study, we compared the traditional SLR process executed accordingly with Cochrane guidelines, with an AI-assisted approach using ChatGPT across various stages, from research question formulation to report writing. Effectiveness was assessed through quantitative measurements of time savings at each phase. Results showed substantial time reductions in several operational tasks, including Gantt chart, generating search terms and suggesting selection criteria. However, critical issues arose in stages requiring interpretative judgement, such as analyzing results, assessing risk of bias and final drafting. While AI cannot replace the role of the researcher, it is a valuable tool to optimize SLR workflow. The combination of human expertise and LLM capabilities presents a promising solution, provided it is accompanied by continuous development of AI systems to improve their reliability, transparency and interoperability."
W4399210979,,10.1007/978-3-031-63028-6_6,2024,Preliminary Systematic Review of Open-Source Large Language Models in Education,Michael Pin-Chuan Lin,4,Lecture notes in computer science,12,
W4392863142,,10.3352/jeehp.2024.21.6,2024,"Opportunities, challenges, and future directions of large language models, including ChatGPT in medical education: a systematic scoping review",Xiaojun Xu,3,Journal of Educational Evaluation for Health Professions,165,"BACKGROUND: ChatGPT is a large language model (LLM) based on artificial intelligence (AI) capable of responding in multiple languages and generating nuanced and highly complex responses. While ChatGPT holds promising applications in medical education, its limitations and potential risks cannot be ignored. METHODS: A scoping review was conducted for English articles discussing ChatGPT in the context of medical education published after 2022. A literature search was performed using PubMed/MEDLINE, Embase, and Web of Science databases, and information was extracted from the relevant studies that were ultimately included. RESULTS: ChatGPT exhibits various potential applications in medical education, such as providing personalized learning plans and materials, creating clinical practice simulation scenarios, and assisting in writing articles. However, challenges associated with academic integrity, data accuracy, and potential harm to learning were also highlighted in the literature. The paper emphasizes certain recommendations for using ChatGPT, including the establishment of guidelines. Based on the review, 3 key research areas were proposed: cultivating the ability of medical students to use ChatGPT correctly, integrating ChatGPT into teaching activities and processes, and proposing standards for the use of AI by medical students. CONCLUSION: ChatGPT has the potential to transform medical education, but careful consideration is required for its full integration. To harness the full potential of ChatGPT in medical education, attention should not only be given to the capabilities of AI but also to its impact on students and teachers."
W4415396422,,10.1002/cesm.70042,2025,Enhancing Evidence Synthesis Efficiency: Leveraging Large Language Models and Agentic Workflows for Optimized Literature Screening,Bing Hu,11,Cochrane Evidence Synthesis and Methods,4,"Background: Public health events of international concern highlight the need for up-to-date evidence curated using sustainable processes that are accessible. In development of the Global Repository of Epidemiological Parameters (grEPI) we explore the performance of an agentic-AI assisted pipeline (GREP-Agent) for screening evidence which capitalizes on recent advancements in large language models (LLMs). Methods: In this study, the performance of the GREP-Agent was evaluated on a data set of 2000 citations from a systematic review on measles using four LLMs (GPT4o, GPT4o-mini, Llama3.1, and Phi4). The GREP-Agent framework integrates multiple LLMs and human feedback to fine-tune its performance, optimize workload reduction and accuracy in screening research articles. The impact on performance of each part of this Agentic-AI system is presented and measured by accuracy, precision, recall, and F1-score metrics. Results: The results show how each phase of the GREP-Agent system incrementally improves accuracy regardless of the LLM. We found that GREP-Agent was able to increase sensitivity across a broad range of open source and proprietary LLMs to 84.2%-88.9% after fine-tuning and to 86.4%-95.3% by varying workload reduction strategies. Performance was significantly impacted by the clarity of the screening questions and setting thresholds for optimized workload reduction strategies. Conclusions: The GREP-Agent shows promise in improving the efficiency and effectiveness of evidence synthesis in dynamic public health contexts. Further development and refinement of adaptable human-in-the-loop AI systems for screening literature are essential to support future public health response activities, while maintaining a human-centric approach."
W4411085860,,10.1007/s00405-025-09504-8,2025,Clinical decision support using large language models in otolaryngology: a systematic review,Rania Filali Ansary,2,European Archives of Oto-Rhino-Laryngology,18,
W7135218648,,10.5281/zenodo.18998810,2026,"EXPLORING HALLUCINATIONS IN LARGE LANGUAGE MODELS (LLMs): A SYSTEMATIC REVIEW OF TYPOLOGIES, ORIGINS, AND MITIGATION APPROACHES",AMAL ALTALHI,2,Journal of Tianjin University Science and Technology,0,"Abstract This systematic literature review pull out the understanding of large language models (LLMs) by thoroughly examining hallucination situations, including the types, causes, and reduce approaches to enhance LLM usefulness in natural language processing (NLP). Electronic databases (Web of Science, IEEE Xplore, Open Review, Google Scholar) were queried by a comprehensive search, generating 1136 records. Of these, 27 met the inclusion criteria and were included. A meta-aggregative approach was used to analyze and synthesize the articles. The research questions formed significant themes for organizing the findings and results section. LLM's ordinary taxonomy includes fact hallucination, honesty hallucination, lack of alignment, conflict in ideas, nonsensical hallucination, random hallucination, object hallucination, and intrinsic and external hallucination. Hallucination causes were training data issues, model limitation/ overfitting, limited context window/ knowledge cutoff, and nuanced language understanding. Effective mitigative approaches were domain-specific fine-tuning, prompting, model reprogramming, and grounding."
W4403915968,,10.31219/osf.io/wm2av,2024,Accelerating Systematic Reviews with Large Language Models: Current Practices and Recommendations,Bo HU,2,,3,"This study investigates the application of Large Language Models (LLMs) in systematic reviews, emphasizing their performance, consistency, and potential for cost efficiency. Through a systematic lineage search, we included 76 studies from 734 articles. The findings reveal that LLMs demonstrate moderate to high performance in title and abstract screening, full-text screening, and data extraction. However, their performance is notably unstable in the stages of literature search and quality assessment. Prompt design emerges as a crucial factor, with Chain of Thought (CoT) prompts frequently enhancing results. While LLMs exhibit moderate to high agreement with human reviewers in some stages, their consistency in quality assessment remains comparatively lower. The research suggests that while LLMs cannot fully replace human reviewers, they serve as valuable assistants in systematic reviews, especially in reducing time and effort. The study also provides practical recommendations for integrating LLMs effectively and discusses the challenges and future research directions in this evolving field."
W7171408732,,10.15173/mumj.v22i2.4166,2026,Performance of large language models on screening titles and abstracts of urology-related systematic reviews,Ata B. Gungor,4,McMaster University Medical Journal,0,"Background: Systematic reviews summarize research evidence to inform clinical practice guidelines and health policy, but the review process is labour-intensive. Authors often screen thousands of abstracts to identify relevant studies to include in their review. Large Language Models (LLMs) may improve title and abstract screening efficiency, but their performance compared to humans require evaluation. Objective: To assess whether locally deployed LLMs accurately screen titles and abstracts of urology-related systematic reviews. Methods: We used LLMs to screen titles and abstracts from three Cochrane reviews in urology totaling 8020 records. Records were screened using predefined criteria, as stated in the Cochrane manuscripts, using three strategies: Qwen3 30B A3B Instruct 2507 alone, Ministral 3 14B Instruct 2512 alone, and their ensemble. A “first-ahead-by-k” approach (k = 2) was used with two models in parallel. We compared their outputs, and once a classification led by a margin of K (i.e., it appeared two more times than any alternative in the running tally) we selected that label as the consensus. Strategy performance metrics were calculated against the human-defined gold standard. Results: Specificity consistently exceeded 90% across strategies (Figure 1). Pooled estimates of sensitivity across the three reviews for strategies were: Ministral 3 14B, 71.1% (95% CI, 64.9-76.6); Qwen3 30B A3B, 53.1% (46.6-59.4); and their ensemble, 72.2% (66.1-77.7). Conclusion: Locally deployed LLMs can screen titles and abstracts with moderate accuracy. Next steps to improve LLM performance include creating random subsamples from each systematic review to perform calibration screening runs with the models, reviewing discrepancies between the model and human-defined gold standard to determine why the performance was variable, and retrieving the exact title and abstract screening criteria used by the authors."
W4413051957,,10.1038/s41746-025-01840-7,2025,Accelerating clinical evidence synthesis with large language models,Zifeng Wang,6,npj Digital Medicine,68,"Clinical evidence synthesis largely relies on systematic reviews (SR) of clinical studies from medical literature. Here, we propose a generative artificial intelligence (AI) pipeline named TrialMind to streamline study search, study screening, and data extraction tasks in SR. We chose published SRs to build TrialReviewBench, which contains 100 SRs and 2,220 clinical studies. For study search, it achieves high recall rates (Ours 0.711-0.834 v.s. Human baseline 0.138-0.232). For study screening, TrialMind beats previous document ranking methods in a 1.5-2.6 fold change. For data extraction, it outperforms a GPT-4's accuracy by 16-32%. In a pilot study, human-AI collaboration with TrialMind improved recall by 71.4% and reduced screening time by 44.2%, while in data extraction, accuracy increased by 23.5% with a 63.4% time reduction. Medical experts preferred TrialMind's synthesized evidence over GPT-4's in 62.5%-100% of cases. These findings show the promise of accelerating clinical evidence synthesis driven by human-AI collaboration."
W7167031171,,10.3233/shti260813,2026,Exploratory Evaluation of Large Language Models for Reducing Language Bias in Systematic Review Screening,Junki Ikeguchi,3,Studies in health technology and informatics,0,"Language bias arises in systematic reviews when non-English studies are excluded owing to resource constraints. Large language models (LLMs) can mitigate this problem through multilingual processing. To assess whether direct multilingual LLM processing reduces language-based disparities in systematic review screening performance compared to translation-mediated approaches. Six state-of-the-art LLMs were evaluated under three conditions: (1) an English benchmark dataset (n = 2,911), (2) direct screening of non-English abstracts (n = 483), and (3) screening of machine-translated non-English abstracts. Performance was measured using sensitivity, specificity, F1 score, balanced accuracy, and workload reduction. All models achieved high sensitivity on English data (≥0.938). Translation-mediated screening substantially reduced sensitivity in some models (range: 0.47-0.54), whereas direct multilingual processing maintained high sensitivity (range: 0.71-1.00). Considerable differences were observed among models. Direct multilingual LLM screening may reduce language-related sensitivity disparities; however, the effects on downstream meta-analytic bias require further investigation."
W7218139948,,,2025,Analysis of the Effectiveness of Large Language Models in Industrial Knowledge Management: A Systematic Literature Review,Jeronimo Malaquias,1,Jurnal Mekintek Jurnal Mekanikal Energi Industri Dan Teknologi,0,"The increasing adoption of Large Language Models (LLMs) has transformed the way organizations manage, access, and utilize knowledge in industrial environments. As industries continue to generate vast amounts of information, LLMs have emerged as powerful tools for enhancing knowledge management processes through advanced natural language understanding, information retrieval, and intelligent decision support capabilities. This study aims to analyze the effectiveness of LLMs in supporting industrial knowledge management and to evaluate their contributions, benefits, and challenges across organizational contexts. A Systematic Literature Review (SLR) approach was employed to examine relevant studies published between 2020 and 2026, with data collected from major scientific databases, including Scopus, Web of Science, IEEE Xplore, and ScienceDirect. The selected literature was analyzed using thematic, content, and comparative analysis techniques to identify patterns, applications, and implementation outcomes. The findings indicate that LLMs significantly enhance knowledge creation through automated report generation, technical documentation support, and lessons-learned extraction. Furthermore, LLM adoption contributes to increased organizational productivity by reducing information search time, supporting decision-making, and improving employee access to critical knowledge resources. However, several challenges remain, including hallucination, data inconsistency, model bias, integration complexity, security and privacy concerns, and issues related to transparency, accountability, and explainability. To maximize their benefits, organizations should implement robust AI governance frameworks, adopt secure knowledge retrieval architectures, and invest in employee AI literacy and training programs. Future research should focus on real-world industrial evaluations, comparative analyses of LLM platforms, and long-term assessments of organizational impacts."
W4403973695,,10.1101/2024.10.30.24316199,2024,Clinical Applications and Limitations of Large Language Models in Nephrology: A Systematic Review,Zoe Unger,6,medRxiv,2,"Abstract Background Large Language Models (LLMs) are emerging as promising tools in healthcare. This systematic review examines LLMs’ potential applications in nephrology, highlighting their benefits and limitations. Methods We conducted a literature search in PubMed and Web of Science, selecting studies based on Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) guidelines. The review focuses on the latest advancements of LLMs in nephrology from 2020 to 2024. PROSPERO registration number: CRD42024550169. Results Fourteen studies met the inclusion criteria and were categorized into five key areas of nephrology: Streamlining workflow, disease prediction and prognosis, laboratory data interpretation and management, renal dietary management, and patient education. LLMs showed high performance in various clinical tasks, including managing continuous renal replacement therapy (CRRT) alarms (GPT-4 accuracy 90-94%) for reducing intensive care unit (ICU) alarm fatigue, and predicting chronic kidney diseases (CKD) progression (improved positive predictive value from 6.7% to 20.9%). In patient education, GPT-4 excelled at simplifying medical information by reducing readability complexity, and accurately translating kidney transplant resources. Gemini provided the most accurate responses to frequently asked questions (FAQs) about CKD. Conclusions While the incorporation of LLMs in nephrology shows promise across various levels of patient care, their broad implementation is still premature. Further research is required to validate these tools in terms of accuracy, rare and critical conditions, and real-world performance."
W7214983381,,10.1016/j.jss.2026.113129,2026,Large language models in software project management: A two-phase systematic literature review,Muhammad Saad Amin,4,Journal of Systems and Software,0,"Background Large Language Models (LLMs) are increasingly integrated into Software Engineering (SE) processes and practices. However, their application within Software Project Management (SPM) remains fragmented and insufficiently synthesized. This absence of systematic evidence prevents both researchers and practitioners from understanding where LLMs can meaningfully contribute to project management processes in relation to established standards such as the Project Management Body of Knowledge (PMBOK). Objectives This study aims to systematically synthesize existing evidence on the application and integration of LLMs in SPM by means of a two-phase method where Phase I systematically searches for existing secondary studies in the topic of interest, and Phase II (whenever applicable) carries out an SLR in the topic of interest. Method We conducted a two-phase method: Phase I is a tertiary study that used 14 research questions to assess the existence, coverage, and quality of prior secondary studies in LLMs applied to SPM. As no suitable SLR was identified from Phase I, we also conducted Phase II, which employed the SEGRESS guidelines to analyze primary studies that empirically evaluated the use of LLMs in SPM via five research questions. Phase II also assessed coverage across PMBOK processes and synthesized reported mechanisms, outcomes, research designs and evaluation approaches. Results The SLR (Phase II) shows that studies employ encoder-only and decoder-only LLM architectures in equal proportion (17 studies each). Their applications focus on a narrow set of SPM tasks, including effort estimation, project planning, and requirements-related activities. Coverage across PMBOK processes is uneven, with limited attention to areas such as procurement, stakeholder, quality, and cost management. Most studies rely on proof-of-concept evaluations, with scarce industrial validation and heterogeneous evaluation metrics. Conclusion The results indicate that current research on LLMs in SPM provides limited guidance for systematic adoption. One of the main drawbacks for such lack of guidance is the weak empirical validation in many of the included primary studies, limited PMBOK coverage, and lack of context-aware integration of LLMs in SPM. 1"
W7136439950,,10.1186/s12873-026-01521-y,2026,Accuracy of the large language model ChatGPT in adult emergency department triage: a systematic review and meta-analysis,Shuang Gao,6,BMC Emergency Medicine,3,"AIM: This study employs a systematic review and meta-analysis to evaluate the accuracy and sensitivity (i.e., correct identification rate of high-risk patients at levels 1–2) of ChatGPT in adult emergency triage and to compare the potential of its different versions. METHODS: A systematic search of English and Chinese medical databases (2022–2025) was performed to identify studies directly comparing the triage performance of ChatGPT with that of human experts or a gold standard. The included studies comprised both simulated case scenarios and real-world clinical data from emergency departments. Literature screening adhered to the PICO framework. Methodological quality was evaluated using the QUADAS-2 tool. Data on overall accuracy and sensitivity were extracted. Meta-analysis was conducted in RevMan 5.4, with a predefined subgroup analysis based on the ChatGPT version. RESULTS: A total of 15 studies were included. The pooled accuracy was 0.51 (95% CI: 0.48–0.55) for ChatGPT-3.5, 0.70 (95% CI: 0.58–0.81) for ChatGPT-4 and its variants, and 0.81 for the optimized versions of ChatGPT (with the 95% confidence interval upper bound approaching 1.0 due to high heterogeneity). No significant differences were observed between ChatGPT and human triage in overall accuracy or sensitivity overall. Subgroup analyses showed that ChatGPT-3.5 performed similarly to humans on both metrics. In contrast, ChatGPT-4 and its variants demonstrated higher accuracy (RR = 1.04, P < 0.001) and sensitivity (RR = 1.46, P = 0.04) compared to human triage, but these advantages were not robust-sensitivity analyses revealed that the statistical significance depended on the inclusion or exclusion of specific studies. CONCLUSION: ChatGPT, particularly ChatGPT-4 and its variants, shows promising capability in identifying high-risk cases during triage, and improvements in version and model optimization are associated with enhanced performance. Nevertheless, this finding is not robust, and the evidence supporting the superiority of ChatGPT-4 remains fragile. Consequently, clinical application should be approached with caution and tailored to specific contexts, with ChatGPT serving as an auxiliary tool."
W4412432132,,10.1016/j.jval.2025.04.1191,2025,MSR39 Large Language Model and Contextual Prioritization of Titles and Abstracts for Systematic Review Screening,Riaz Qureshi,2,Value in Health,0,
W4410251088,,10.1186/s12874-025-02583-5,2025,Streamlining systematic reviews with large language models using prompt engineering and retrieval augmented generation,Fouad S. Trad,7,BMC Medical Research Methodology,43,"BACKGROUND: Systematic reviews (SRs) are essential to formulate evidence-based guidelines but require time-consuming and costly literature screening. Large Language Models (LLMs) can be a powerful tool to expedite SRs. METHODS: We conducted a comparative study to evaluate the performance of a commercial tool, Rayyan, and an in-house LLM-based system in automating the screening of a completed SR on Vitamin D and falls. The SR retrieved 14,439 articles, and Rayyan was trained with 2,000 manually screened articles to categorize the rest as most likely to exclude/include, likely to exclude/include and undecided. We analyzed Rayyan's title/abstract screening performance using different inclusion thresholds. For the LLM, we used prompt engineering for title/abstract screening and Retrieval-Augmented Generation (RAG) for full-text screening. We evaluated performance using article exclusion rate (AER), false negative rate (FNR), specificity, positive predictive value (PPV), and negative predictive value (NPV). Additionally, we compared the time required to complete screening steps of the SR using both approaches against the manual screening method. RESULTS: Using Rayyan, including considered as undecided or likely to include for title/abstract screening resulted in an AER of 72.1% and an FNR of 5%. The total estimated screening time, including manual review of articles flagged by Rayyan, was 54.7 hours. Lowering the Rayyan threshold to 'likely to exclude' reduced the FNR to 0% and the AER to 50.7%, but increased the screening time to 81.3 h. Using the LLM system, after title/abstract and full-text screening, 78 articles remained for manual review, including all 20 identified by traditional methods. The LLM achieved an AER of 99.5%, specificity of 99.6%, PPV of 25.6%, and NPV of 100%, with a total screening time of 25.5 h, including manual review of the 78 articles, reducing the manual screening time by 95.5%. CONCLUSIONS: The LLM-based system significantly enhances SR efficiency, compared to manual methods and Rayyan while maintaining low FNR."
W4415757150,,10.2139/ssrn.5688905,2025,Systematic Review of Large Language Model Applications in Transport Engineering,Sneharsh Belsare,4,SSRN Electronic Journal,0,
W4417479740,,10.1016/j.jval.2025.09.2960,2025,SA18 Calibrating Large Language Model Probabilities to Achieve Target Recall in Systematic Review Screening,Ṣẹ̀yẹ Abògúnr̀in,3,Value in Health,0,
W4406849765,,10.3389/fdata.2024.1505284,2025,On explaining recommendations with Large Language Models: a review,Alan Said,1,Frontiers in Big Data,34,"The rise of Large Language Models (LLMs), such as LLaMA and ChatGPT, has opened new opportunities for enhancing recommender systems through improved explainability. This paper provides a systematic literature review focused on leveraging LLMs to generate explanations for recommendations-a critical aspect for fostering transparency and user trust. We conducted a comprehensive search within the ACM Guide to Computing Literature, covering publications from the launch of ChatGPT (November 2022) to the present (November 2024). Our search yielded 232 articles, but after applying inclusion criteria, only six were identified as directly addressing the use of LLMs in explaining recommendations. This scarcity highlights that, despite the rise of LLMs, their application in explainable recommender systems is still in an early stage. We analyze these select studies to understand current methodologies, identify challenges, and suggest directions for future research. Our findings underscore the potential of LLMs improving explanations of recommender systems and encourage the development of more transparent and user-centric recommendation explanation solutions."
W7125684565,,10.2139/ssrn.6086099,2026,Systematic Review of Large Language Model Applications in Transport Engineering,Ashish Bhaskar,4,SSRN Electronic Journal,0,
W4406421570,,10.1093/jamia/ocaf008,2025,"Improving large language model applications in biomedicine with retrieval-augmented generation: a systematic review, meta-analysis, and clinical development guidelines",Siru Liu,3,Journal of the American Medical Informatics Association,197,"OBJECTIVE: The objectives of this study are to synthesize findings from recent research of retrieval-augmented generation (RAG) and large language models (LLMs) in biomedicine and provide clinical development guidelines to improve effectiveness. MATERIALS AND METHODS: We conducted a systematic literature review and a meta-analysis. The report was created in adherence to the Preferred Reporting Items for Systematic Reviews and Meta-Analyses 2020 analysis. Searches were performed in 3 databases (PubMed, Embase, PsycINFO) using terms related to ""retrieval augmented generation"" and ""large language model,"" for articles published in 2023 and 2024. We selected studies that compared baseline LLM performance with RAG performance. We developed a random-effect meta-analysis model, using odds ratio as the effect size. RESULTS: Among 335 studies, 20 were included in this literature review. The pooled effect size was 1.35, with a 95% confidence interval of 1.19-1.53, indicating a statistically significant effect (P = .001). We reported clinical tasks, baseline LLMs, retrieval sources and strategies, as well as evaluation methods. DISCUSSION: Building on our literature review, we developed Guidelines for Unified Implementation and Development of Enhanced LLM Applications with RAG in Clinical Settings to inform clinical applications using RAG. CONCLUSION: Overall, RAG implementation showed a 1.35 odds ratio increase in performance compared to baseline LLMs. Future research should focus on (1) system-level enhancement: the combination of RAG and agent, (2) knowledge-level enhancement: deep integration of knowledge into LLM, and (3) integration-level enhancement: integrating RAG systems within electronic health records."
W4399702760,,10.1186/s13643-024-02575-4,2024,Title and abstract screening for literature reviews using large language models: an exploratory study in the biomedical domain,Fabio Dennstädt,5,Systematic Reviews,151,"BACKGROUND: Systematically screening published literature to determine the relevant publications to synthesize in a review is a time-consuming and difficult task. Large language models (LLMs) are an emerging technology with promising capabilities for the automation of language-related tasks that may be useful for such a purpose. METHODS: LLMs were used as part of an automated system to evaluate the relevance of publications to a certain topic based on defined criteria and based on the title and abstract of each publication. A Python script was created to generate structured prompts consisting of text strings for instruction, title, abstract, and relevant criteria to be provided to an LLM. The relevance of a publication was evaluated by the LLM on a Likert scale (low relevance to high relevance). By specifying a threshold, different classifiers for inclusion/exclusion of publications could then be defined. The approach was used with four different openly available LLMs on ten published data sets of biomedical literature reviews and on a newly human-created data set for a hypothetical new systematic literature review. RESULTS: The performance of the classifiers varied depending on the LLM being used and on the data set analyzed. Regarding sensitivity/specificity, the classifiers yielded 94.48%/31.78% for the FlanT5 model, 97.58%/19.12% for the OpenHermes-NeuralChat model, 81.93%/75.19% for the Mixtral model and 97.58%/38.34% for the Platypus 2 model on the ten published data sets. The same classifiers yielded 100% sensitivity at a specificity of 12.58%, 4.54%, 62.47%, and 24.74% on the newly created data set. Changing the standard settings of the approach (minor adaption of instruction prompt and/or changing the range of the Likert scale from 1-5 to 1-10) had a considerable impact on the performance. CONCLUSIONS: LLMs can be used to evaluate the relevance of scientific publications to a certain review topic and classifiers based on such an approach show some promising results. To date, little is known about how well such systems would perform if used prospectively when conducting systematic literature reviews and what further implications this might have. However, it is likely that in the future researchers will increasingly use LLMs for evaluating and classifying scientific publications."
W4416403111,,10.3389/fmed.2025.1670824,2025,Clinical applications of large language models in knee osteoarthritis: a systematic review,Zebing Ma,7,Frontiers in Medicine,6,"Background and aims: Knee osteoarthritis (KOA) is a common chronic degenerative disease that significantly impacts patients' quality of life. With the rapid advancement of artificial intelligence, large language models (LLMs) have demonstrated potential in supporting medical information extraction, clinical decision-making, and patient education through their natural language processing capabilities. However, the current landscape of LLM applications in the KOA domain, along with their methodological quality, has yet to be systematically reviewed. Therefore, this systematic review aims to comprehensively summarize existing clinical studies on LLMs in KOA, evaluate their performance and methodological rigor, and identify current challenges and future research directions. Methods: Following the PRISMA guidelines, a systematic search was conducted in PubMed, Cochrane Library, Embase databases and Web of science for literature published up to June 2025. The protocol was preregistered on the OSF platform. Studies were screened using standardized inclusion and exclusion criteria. Key study characteristics and performance evaluation metrics were extracted. Methodological quality was assessed using tools such as Cochrane RoB, STROBE, STARD, and DISCERN. Additionally, the CLEAR-LLM and CliMA-10 frameworks were applied to provide complementary evaluations of quality and performance. Results: A total of 16 studies were included, covering various LLMs such as ChatGPT, Gemini, and Claude. Application scenarios encompassed text generation, imaging diagnostics, and patient education. Most studies were observational in nature, and overall methodological quality ranged from moderate to high. Based on CliMA-10 scores, LLMs exhibited upper-moderate performance in KOA-related tasks. The ChatGPT-4 series consistently outperformed other models, especially in structured output generation, interpretation of clinical terminology, and content accuracy. Key limitations included insufficient sample representativeness, inconsistent control over hallucinated content, and the lack of standardized evaluation tools. Conclusion: Large language models show notable potential in the KOA field, but their clinical application is still exploratory and limited by issues such as sample bias and methodological heterogeneity. Model performance varies across tasks, underscoring the need for improved prompt design and standardized evaluation frameworks. With real-world data and ethical oversight, LLMs may contribute more significantly to personalized KOA management. Systematic review registration: https://osf.io/jy4kz, identifier 10.17605/OSF.IO/479R8."
W7166720108,,10.1192/bjo.2026.11144,2026,Large Language Models in Dementia Assessment and Management: A Systematic Review,Ami Mehta,5,BJPsych Open,0,"Aims: Large language models (LLMs) are increasingly proposed to support healthcare delivery, yet their ability to address the distinctive challenges of dementia care remains unclear. Dementia is characterised by progressive cognitive impairment, communication barriers, high caregiver burden, and substantial demands on memory services and old age psychiatry. We systematically reviewed and classified empirical applications of LLMs in dementia assessment and care, and critically appraised the maturity of evidence, intended users, and reporting of safety, fairness, and governance relevant to clinical implementation. Methods: We searched EMBASE, MEDLINE, PsycINFO, PubMed, the ACL Anthology, the ACM Digital Library, arXiv, medRxiv, and bioRxiv (2017–June 2025). Eligible studies were empirical evaluations (simulation-based or experimental) of LLMs used for dementia-relevant clinical, caregiver-facing, or research tasks; non-evaluative commentaries were excluded. Titles/abstracts were screened by multiple reviewers, with full-text screeningconducted in accordance with PRISMA; disagreements were resolved by discussion. Using a standardised extraction form, we recorded study design, model attributes, input modalities, dementia-related use cases, evaluation setting (benchmark/vignette vs clinical data), outcomes, and reporting of ethics approval, bias assessment, and hallucination-related issues. Findings were synthesised narratively and grouped using a predefined taxonomy of clinical, patient-facing, and research-oriented applications. Results: We identified 35 dementia-focused studies, several covering multiple applications. The most frequently reported functions were screening/early detection (26/35) and diagnostic support (28/35), typically framed as classification tasks. Other common applications included research facilitation (19/35), evaluation/outcome measurement (19/35), information extraction (14/35), and prediction/risk stratification (13/35). Multimodal approaches were reported in (14/35). Intended users were most often researchers (22/35) and clinicians (20/35), with fewer studies targeting people living with dementia (10/35) or caregivers. Inputs included speech/audio (20/35) and free text (18/35); outputs most commonly included classification labels (26/35) and risk scores (11/35). Performance metrics were reported in 18/35. Reporting of safeguards was inconsistent: ethics approval (10/35, 29%), bias assessment (10/35, 29%), and explicit hallucination reporting (5/35, 14%). Conclusion: LLMs show promising experimental performance for dementia-related classification and early detection tasks, suggesting potential roles in augmenting assessment workflows. However, evidence is largely preclinical, with limited real-world validation and inconsistent reporting of fairness, safety, and governance. Future research should prioritise prospective evaluation in memory services, robust bias and hallucination assessment, and co-designed implementation with people living with dementia and caregivers to support safe, equitable integration in psychiatric practice."
W4406769232,,10.3389/fmed.2024.1512824,2025,Emerging applications of NLP and large language models in gastroenterology and hepatology: a systematic review,Mahmud Omar,6,Frontiers in Medicine,15,"Background and aim: In the last years, natural language processing (NLP) has transformed significantly with the introduction of large language models (LLM). This review updates on NLP and LLM applications and challenges in gastroenterology and hepatology. Methods: Registered with PROSPERO (CRD42024542275) and adhering to PRISMA guidelines, we searched six databases for relevant studies published from 2003 to 2024, ultimately including 57 studies. Results: Our review of 57 studies notes an increase in relevant publications in 2023-2024 compared to previous years, reflecting growing interest in newer models such as GPT-3 and GPT-4. The results demonstrate that NLP models have enhanced data extraction from electronic health records and other unstructured medical data sources. Key findings include high precision in identifying disease characteristics from unstructured reports and ongoing improvement in clinical decision-making. Risk of bias assessments using ROBINS-I, QUADAS-2, and PROBAST tools confirmed the methodological robustness of the included studies. Conclusion: NLP and LLMs can enhance diagnosis and treatment in gastroenterology and hepatology. They enable extraction of data from unstructured medical records, such as endoscopy reports and patient notes, and for enhancing clinical decision-making. Despite these advancements, integrating these tools into routine practice is still challenging. Future work should prospectively demonstrate real-world value."
W7135154108,,10.1016/j.prosdent.2026.02.009,2026,Performance of large language models conducting systematic review tasks in prosthodontics,Rata Rokhshad,5,Journal of Prosthetic Dentistry,1,
W7167597891,,10.1021/acs.jpcc.6c03290,2026,Literature Screening and Knowledge Extraction via Domain-Adaptive Transfer Learning with Large Language Models,Xitao Wang,5,The Journal of Physical Chemistry C,0,"The rapid growth of scientific literature poses major challenges for large-scale knowledge extraction in materials science, particularly when domain-specific concepts are expressed ambiguously or require deep contextual reasoning. Here, we propose a workflow that integrates domain-adaptive transfer learning, knowledge distillation, and reinforcement learning for high-precision literature screening and structured knowledge extraction in a ferroelectric Curie-temperature case study. A large open-source language model is used as a teacher to generate high-fidelity pseudolabels from full-text articles, which are then employed to train a compact student model via reinforcement-learning-based optimization. Under the same title-and-abstract input condition, the student model achieves 93.6% classification accuracy, exceeding the zero-shot Qwen3-235B-A22B baseline by 7.1 percentage points, while reducing parameter count by 138-fold and inference time by approximately 50-fold relative to the full-text teacher pipeline. This study clarifies the contextual ambiguity of “Curie temperature” in ferroelectric versus ferromagnetic literature and supports the construction of a verified ferroelectric materials knowledge base from the studied corpus. 5-fold cross-validation and ablation studies confirm the effectiveness of the proposed workflow for this task. These results demonstrate a task-specific route for combining full-text pseudolabeling with lightweight title-and-abstract screening while retaining full-text verification before database construction."
W4409830389,,10.1089/lap.2024.0277,2025,The Role of Artificial Intelligence Large Language Models in Literature Search Assistance to Evaluate Inguinal Hernia Repair Approaches,João Pedro Gonçalves Kasakewitch,8,Journal of Laparoendoscopic & Advanced Surgical Techniques,3,"Aim: This study assesses the reliability of artificial intelligence (AI) large language models (LLMs) in identifying relevant literature comparing inguinal hernia repair techniques. Material and Methods: We used LLM chatbots (Bing Chat AI, ChatGPT versions 3.5 and 4.0, and Gemini) to find comparative studies and randomized controlled trials on inguinal hernia repair techniques. The results were then compared with existing systematic reviews (SRs) and meta-analyses and checked for the authenticity of listed articles. Results: LLMs screened 22 studies from 2006 to 2023 across eight journals, while the SRs encompassed a total of 42 studies. Through thorough external validation, 63.6% of the studies (14 out of 22), including 10 identified through Chat GPT 4.0 and 6 via Bing AI (with an overlap of 2 studies between them), were confirmed to be authentic. Conversely, 36.3% (8 out of 22) were revealed as fabrications by Google Gemini (Bard), with two (25.0%) of these fabrications mistakenly linked to valid DOIs. Four (25.6%) of the 14 real studies were acknowledged in the SRs, which represents 18.1% of all LLM-generated studies. LLMs missed a total of 38 (90.5%) of the studies included in the previous SRs, while 10 real studies were found by the LLMs but were not included in the previous SRs. Between those 10 studies, 6 were reviews, and 1 was published after the SRs, leaving a total of three comparative studies missed by the reviews. Conclusions: This study reveals the mixed reliability of AI language models in scientific searches. Emphasizing a cautious application of AI in academia and the importance of continuous evaluation of AI tools in scientific investigations."
W7128197893,,10.1093/biomethods/bpag006,2026,Large language model-based multiagent collaboration for abstract screening toward automated systematic reviews,Opeoluwa Akinseloyin,3,Biology Methods and Protocols,5,"Systematic reviews (SRs) are essential for evidence-based practice but remain labor-intensive, especially during abstract screening. This study evaluates whether multiple large language models (multi-LLMs) collaboration can improve the efficiency and reduce costs for abstract screening. Abstract screening was framed as a question-answering (QA) task using cost-effective LLMs. Three multi-LLM collaboration strategies were evaluated, including majority voting by averaging opinions of peers, multi-agent debate for answer refinement, and LLM-based adjudication against answers of individual QA baselines. These strategies were evaluated on 28 SRs of the CLEF eHealth 2019 technology-assisted review benchmark using standard performance metrics such as mean average precision (MAP) and work saved over sampling at 95% recall (work saved over sampling WSS@95%). Multi-LLM collaboration significantly outperformed QA baselines. Majority voting was overall the best strategy, achieving the highest MAP 0.462 and 0.341 on subsets of SRs about clinical intervention and diagnostic technology assessment, respectively, with WSS@95% 0.606 and 0.680, enabling in theory up to 68% workload reduction at 95% recall of all relevant studies. Multi-agent debate improved weaker models most. Our own adjudicator-as-a-ranker method was the second strongest approach, surpassing adjudicator-as-a-judge, but at a significantly higher cost than majority voting and debating. Multi-LLM collaboration substantially improves abstract screening efficiency, and the success lies in model diversity. Making the best use of diversity, majority voting stands out in terms of both excellent performance and low cost compared to adjudication. Despite context-dependent gains and diminishing model diversity, multi-agent debate is still a cost-effective strategy and a potential direction of further research."
W7125913448,,10.1007/s44163-025-00784-x,2026,Large language models for clinical artificial intelligence in healthcare a systematic review,Rawan Ghnemat,2,Discover Artificial Intelligence,9,"Abstract Large Language Models (LLMs) have demonstrated the capacity to process, reason, and generate extensive volumes of data, providing a novel paradigm for integrating generative artificial intelligence (GenAI) into the medical field. Multimodal LLMs (MLLMs) extend these capabilities by incorporating diverse data modalities into unified representations, including genomics, medical imaging, and clinical text. This systematic review synthesizes advancements from 246 records identified between January 2020 and September 2025, of which 90 studies were included after full-text screening, to address critical gaps in understanding the clinical role of LLM and MLLM in healthcare. We trace the evolution from classical natural language processing (NLP) approaches to modern transformer-based architectures, summarize their technical foundations, and examine their construction, evaluation, and deployment in medical workflows. Key contributions include highlighting multimodal integration (e.g., imaging-genomics-text fusion), ethical governance frameworks, and validated domain-specific fine-tuning in clinical settings. We also highlight advances in Prompting, Retrieval-Augmented Generation (RAG), and Multi-Agent (agentic) workflows, providing a critical assessment of their benefits and limitations. In addition, we analyze challenges such as hallucinations, bias, and privacy risks, while providing actionable guidelines for clinicians, developers, and policymakers to improve regulatory compliance. By consolidating the nomenclature and systematically evaluating GenAI in medicine, this review offers evidence-based recommendations and directions for the safe and effective integration of generative AI into healthcare. The findings are intended as an authoritative guide for researchers and practitioners, bridge principles, clinical applications, and policy considerations for LLM and MLLM."
W4415976151,10.48550/arxiv.2504.02917,10.48550/arxiv.2504.02917,2025,Bias in Large Language Models Across Clinical Applications: A Systematic Review,Thanathip Suenghataiphorn,4,arXiv (Cornell University),8,"Background: Large language models (LLMs) are rapidly being integrated into healthcare, promising to enhance various clinical tasks. However, concerns exist regarding their potential for bias, which could compromise patient care and exacerbate health inequities. This systematic review investigates the prevalence, sources, manifestations, and clinical implications of bias in LLMs. Methods: We conducted a systematic search of PubMed, OVID, and EMBASE from database inception through 2025, for studies evaluating bias in LLMs applied to clinical tasks. We extracted data on LLM type, bias source, bias manifestation, affected attributes, clinical task, evaluation methods, and outcomes. Risk of bias was assessed using a modified ROBINS-I tool. Results: Thirty-eight studies met inclusion criteria, revealing pervasive bias across various LLMs and clinical applications. Both data-related bias (from biased training data) and model-related bias (from model training) were significant contributors. Biases manifested as: allocative harm (e.g., differential treatment recommendations); representational harm (e.g., stereotypical associations, biased image generation); and performance disparities (e.g., variable output quality). These biases affected multiple attributes, most frequently race/ethnicity and gender, but also age, disability, and language. Conclusions: Bias in clinical LLMs is a pervasive and systemic issue, with a potential to lead to misdiagnosis and inappropriate treatment, particularly for marginalized patient populations. Rigorous evaluation of the model is crucial. Furthermore, the development and implementation of effective mitigation strategies, coupled with continuous monitoring in real-world clinical settings, are essential to ensure the safe, equitable, and trustworthy deployment of LLMs in healthcare."
W4414983308,,10.1136/bmjdhai-2025-000017,2025,Large language model enhanced framework for systematic reviews and meta-analyses,Jiashu Shen,6,BMJ Digital Health & AI,2,"Objective: To evaluate and synthesise current applications of large language models (LLMs) in systematic reviews and meta-analyses (SRMAs), identify key limitations and propose an enhanced theoretical framework to improve the efficiency, scalability and reliability of evidence synthesis. Methods and analysis: We conducted a narrative review of recent studies applying LLMs across key SRMA stages. A total of 21 publications were analysed for model type, task application, accuracy metrics and workflow impact. Building on this evidence base, we designed a comprehensive LLM-enhanced SRMA framework that categorises LLM roles as consultants and assistants, integrates human-in-the-loop strategies and uses retrieval-augmented generation (RAG) and agent-based architectures to address critical challenges including hallucinations, bias and workflow inefficiency. Results: The reviewed literature demonstrated that LLMs can support various SRMA tasks with reported accuracy ranging from 61% to 99%, showing particular promise in literature screening and data extraction. Our proposed framework conceptualises modular integration of LLMs across all six SRMA stages, with LLMs serving as consultants for research question formulation and search strategy development and as assistants for task automation including abstract screening and structured data extraction. The framework incorporates RAG technology to reduce hallucinations by grounding outputs in retrieved literature and employs agent-based orchestration for complex analytical workflows. Theoretical analysis suggests potential for significant efficiency gains while maintaining methodological rigour through strategic human oversight. Conclusion: LLMs offer substantial theoretical potential to transform evidence synthesis by improving efficiency, scalability and consistency across SRMA workflows. The proposed LLM-enhanced framework provides a systematic, theoretically grounded approach for integrating advanced artificial intelligence capabilities into existing SRMA methodologies while preserving essential human oversight and analytical integrity. Future empirical studies are needed to validate the framework's practical effectiveness, establish implementation protocols and demonstrate real-world benefits in evidence-based medicine."
W4408987202,,10.1101/2025.03.28.25324834,2025,Prompt Engineering in Large Language Models for Patient Education: A Systematic Review,Aya Mudrik,5,medRxiv,7,"ABSTRACT Background Large language models (LLMs) have shown promise in generating patient-friendly medical content, but their outputs often vary in accuracy, readability, and relevance. Prompt engineering—structuring inputs to guide LLM responses—may improve the quality of educational materials, yet its impact on patient education remains unclear. Objectives To systematically review whether prompt engineering improves readability, accuracy, and usability of LLM-generated content for patient education. Methods We conducted a systematic review in accordance with PRISMA guidelines. PubMed, Scopus, and Web of Science were searched for original studies evaluating prompt engineering techniques in patient education. Data were extracted on prompt types, LLM models used, and outcomes. Risk of bias was assessed using the QUADAS-2 tool, and a narrative synthesis was performed. Results Our search identified five studies that met our criteria, focusing on answering patient questions and generating medical information. Prompt engineering techniques included instruction-based, elaborated, role-defining, scene-defining, and domain-specific prompts. Structured prompting improved accuracy and comprehensiveness in several cases, particularly when specific formats or custom instructions were used. Readability gains were notable when prompts explicitly requested simpler language and reading levels, though some strategies unintentionally increased complexity. Variability in effectiveness across LLMs and prompt types was observed. Conclusion Prompt engineering can enhance the clarity and, in some cases, the accuracy of LLM-generated patient education materials. However, benefits vary by model and strategy. Standardized approaches and further research are needed to optimize prompts, minimize bias, and support reliable, accessible patient communication."
W7114905629,,10.1093/bib/bbaf631.059,2025,Should we use large language models to perform systematic reviews?,TH Chan,4,Briefings in Bioinformatics,0,"Abstract Background Large language models (LLMs) have shown promising performances in literature search, screening of full-text scientific articles, data extraction, and produce a written summary. LLMs have a strong potential in automating the process of conducting systematic reviews. Nonetheless, conducting systematic reviews is also a training exercise for critical appraisal of scientific evidence and synthesis of scientific claims. Overreliance on LLMs may erode comprehension and analytical skills in reading scientific articles over time, leading to cognitive offloading. Other risks may involve hallucination and responsibility problem. Methods To address both opportunities and limitations of LLMs when applied in systematic reviews, the LLM-Augmented Systematic Evidence Review (LASER) framework is proposed. Results The framework recommends different degree of LLM usage in different steps in the systematic review process to promote efficiency, ensure accuracy, and prevent cognitive offloading of the researcher involved. In this framework, the specific degree of LLM usage may depend on the career stage of the researcher and the accuracy of the LLM."
W4416347706,,10.3352/jeehp.2025.22.36,2025,Performance of large language models in medical licensing examinations: a systematic review and meta-analysis,Haniyeh Nouri,6,Journal of Educational Evaluation for Health Professions,14,"PURPOSE: This study systematically evaluates and compares the performance of large language models (LLMs) in answering medical licensing examination questions. By conducting subgroup analyses based on language, question format, and model type, this meta-analysis aims to provide a comprehensive overview of LLM capabilities in medical education and clinical decision-making. METHODS: This systematic review, registered in PROSPERO and following PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) guidelines, searched MEDLINE (PubMed), Scopus, and Web of Science for relevant articles published up to February 1, 2025. The search strategy included Medical Subject Headings (MeSH) terms and keywords related to (""ChatGPT"" OR ""GPT"" OR ""LLM variants"") AND (""medical licensing exam*"" OR ""medical exam*"" OR ""medical education"" OR ""radiology exam*""). Eligible studies evaluated LLM accuracy on medical licensing examination questions. Pooled accuracy was estimated using a random-effects model, with subgroup analyses by LLM type, language, and question format. Publication bias was assessed using Egger's regression test. RESULTS: This systematic review identified 2,404 studies. After removing duplicates and excluding irrelevant articles through title and abstract screening, 36 studies were included after full-text review. The pooled accuracy was 72% (95% confidence interval, 70.0% to 75.0%) with high heterogeneity (I2=99%, P<0.001). Among LLMs, GPT-4 achieved the highest accuracy (81%), followed by Bing (79%), Claude (74%), Gemini/Bard (70%), and GPT-3.5 (60%) (P=0.001). Performance differences across languages (range, 62% in Polish to 77% in German) were not statistically significant (P=0.170). CONCLUSION: LLMs, particularly GPT-4, can match or exceed medical students' examination performance and may serve as supportive educational tools. However, due to variability and the risk of errors, they should be used cautiously as complements rather than replacements for traditional learning methods."
W4394009806,,10.1016/j.ajic.2024.03.016,2024,Utilizing natural language processing and large language models in the diagnosis and prediction of infectious diseases: A systematic review,Mahmud Omar,4,American Journal of Infection Control,59,
W7222962256,,10.83095/ceurws_04296_paper1,2026,Fine-Tuning A 1.2B Large Language Model for Systematic Review Screening in Educational Technology,Kweku Andoh Yamoah,5,ceur-ws.org,0,
W4413467640,,10.2196/76252,2025,Automated Literature Screening for Hepatocellular Carcinoma Treatment Through Integration of 3 Large Language Models: Methodological Study,Chen Pan,6,JMIR Medical Informatics,6,"BACKGROUND Primary liver cancer, particularly hepatocellular carcinoma (HCC), poses significant clinical challenges due to late-stage diagnosis, tumor heterogeneity, and rapidly evolving therapeutic strategies. While systematic reviews and meta-analyses are essential for updating clinical guidelines, their labor-intensive nature limits timely evidence synthesis. OBJECTIVE This study proposes an automated literature screening workflow powered by large language models (LLMs) to accelerate evidence synthesis for HCC treatment guidelines. METHODS We developed a tripartite LLM framework integrating Doubao-1.5-pro-32k, Deepseek-v3, and DeepSeek-R1-Distill-Qwen-7B to simulate collaborative decision-making for study inclusion and exclusion. The system was evaluated across 9 reconstructed datasets derived from published HCC meta-analyses, with performance assessed using accuracy, agreement metrics (κ and prevalence-adjusted bias-adjusted κ), recall, precision, F1-scores, and computational efficiency parameters (processing time and cost). RESULTS The framework demonstrated good performance, with a weighted accuracy of 0.96 and substantial agreement (prevalence-adjusted bias-adjusted κ=0.91), achieving high weighted recall (0.90) but modest weighted precision (0.15) and F1-scores (0.22). Computational efficiency varied across datasets (processing time: 248-5850 s; cost: US $0.14-$3.68 per dataset). CONCLUSIONS This LLM-driven approach shows promise for accelerating evidence synthesis in HCC care by reducing screening time while maintaining methodological rigor. Key limitations related to clinical context sensitivity and error propagation highlight the need for reinforcement learning integration and domain-specific fine-tuning. LLM agent architectures with reinforcement learning offer a practical path for streamlining guideline updates, though further optimization is needed to improve specialization and reliability in complex clinical settings."
W4386399156,,10.1101/2023.09.03.23294842,2023,How Large Language Models Perform on the United States Medical Licensing Examination: A Systematic Review,Dana Brin,6,medRxiv,16,"ABSTRACT Objective The United States Medical Licensing Examination (USMLE) assesses physicians’ competency and passing is a requirement to practice medicine in the U.S. With the emergence of large language models (LLMs) like ChatGPT and GPT-4, understanding their performance on these exams illuminates their potential in medical education and healthcare. Materials and Methods A literature search following the 2020 PRISMA guidelines was conducted, focusing on studies using official USMLE questions and publicly available LLMs. Results Three relevant studies were found, with GPT-4 showcasing the highest accuracy rates of 80-90% on the USMLE. Open-ended prompts typically outperformed multiple-choice ones, with 5-shot prompting slightly edging out zero-shot. Conclusion LLMs, especially GPT-4, display proficiency in tackling USMLE-standard questions. While the USMLE is a structured evaluation tool, it may not fully capture the expansive capabilities and limitations of LLMs in medical scenarios. As AI integrates further into healthcare, ongoing assessments against trusted benchmarks are essential."
W7223446955,,10.83095/ceurws_04296_001,2026,Fine-Tuning A 1.2B Large Language Model for Systematic Review Screening in Educational Technology,Kweku Andoh Yamoah,5,ceur-ws.org,0,
W7172325417,,10.1017/s0266462326104073,2026,A prompt-based tutorial for large language model–assisted screening in systematic reviews and meta-analyses,Tongyu Qiu,4,International Journal of Technology Assessment in Health Care,0,"OBJECTIVES: Title/abstract and full-text screening are among the most time-consuming stages of systematic reviews. Large language models (LLMs) such as ChatGPT can assist in screening, but prior evaluations have largely focused on abstract-level decisions and require coding or application programming interfaces (API) access, limiting practical application. This study developed and evaluated a structured, prompt-based approach that enables LLMs to perform both abstract and full-text screening without programming or API integration. METHODS: Using datasets from three completed meta-analyses, we implemented a stepwise training framework involving comprehension checks, criterion-specific feedback, and iterative prompt refinement. Study 1 optimized file format and batch size using 1,000 abstracts and 50 full texts. Studies 2 and 3 validated the approach using independent datasets on teletherapy versus in-person therapy (1,321 abstracts, 82 full texts) and emotion regulation and mental health (1,500 abstracts, 129 full texts), respectively. Human reviewers' decisions served as the reference standard. RESULTS: In Study 1, the LLM achieved 98 percent sensitivity and 80.6 percent specificity at abstract screening, with batches of 50 abstracts in plain-text format performing optimally. In Study 2, sensitivity and specificity were 100 percent and 85.6 percent for abstracts, with 82.1 percent full-text accuracy and all eligible studies retained. In Study 3, abstract screening achieved 100 percent sensitivity and 93 percent specificity, while full-text screening achieved 96.2 percent sensitivity and 90.3 percent specificity. CONCLUSIONS: A structured, prompt-based approach enabled LLMs to achieve high sensitivity and specificity across both screening stages without coding or APIs. With human oversight, this accessible workflow can reduce reviewer burden and accelerate evidence synthesis."
W4414532163,,10.1007/s10815-025-03679-z,2025,The potential of large language models in the field of infertility: a systematic review,Wei Li,11,Journal of Assisted Reproduction and Genetics,1,
W7152014256,,10.1371/journal.pdig.0001354,2026,A systematic review of the limitations of large language models in generating healthcare content,Mohsen Khosravi,5,PLOS Digital Health,5,"Large language models (LLMs) have recently gained prominence in healthcare content provision due to their numerous advantages. Despite these benefits, LLMs exhibit notable limitations in this domain. This study aimed to systematically identify the limitations of LLMs in provision of healthcare content. This study was a systematic review conducted in September 2025, including articles published in English between 2018 and 2025. Searches were performed in PubMed, Scopus, and the Cochrane Database of Systematic Reviews. Two independent evaluators screened the references and assessed quality of the selected studies using the Authority, Accuracy, Coverage, Objectivity, Date, and Significance (AACODS) checklist. Data were analyzed using Boyatzis's qualitative thematic approach with an inductive methodology, applying the input-process-output (IPO) model as the analytical framework. A total of 81 studies were included in the final analysis. The included studies were predominantly of high quality and demonstrated minimal risk of bias. The thematic analysis identified key themes: data limitations, dependence on input and prompt quality, accessibility issues, model design and architecture constraints, interaction challenges, response quality and comprehensiveness, and ethical, safety, and regulatory concerns. The study identified multiple limitations of LLMs in healthcare, with output issues being most common. In this regard, the most frequently cited limitation was the accuracy gap. However, these output issues were mainly resulted from flaws in input data, emphasizing the crucial role of input quality. The study also proposed strategies to address these challenges."
W4410421218,,10.1016/j.identj.2025.04.015,2025,Can Large Language Models Serve as Reliable Tools for Information in Dentistry? A Systematic Review,Nora Alhazmi,6,International Dental Journal,24,"Large language models (LLMs) have gained popularity among dental students for generating subject-related answers. However, their widespread use raises significant concerns about misinformation. This systematic review aims to critically evaluate studies assessing the performance of LLMs in dentistry. A comprehensive electronic search was conducted in PubMed/Medline, Scopus, Embase, Web of Science, Google Scholar, and the Saudi Digital Library to identify studies published up to September 2024. The study quality was assessed using the Prediction Model Risk of Bias Assessment Tool (PROBAST). A total of 2030 studies have been identified. After removing 907 duplicate records, 1123 studies remained for screening. Ultimately, 31 studies met the inclusion criteria. Approximately half of these studies were classified as ""high risk,"" while the remainder were classified as ""low risk."" The applicability of the findings was rated as ""low concern."" The primary limitations of LLMs include their inability to specify information sources and their tendency to generate fabricated citations. Based on this review, LLMs hold promise as supplementary educational tools in dentistry. Evidence indicates that students using LLMs may achieve improved academic performance compared to traditional methods. However, concerns about occasional inaccuracies and unreliable citations underscore the need for further research, integration with validated sources, and adherence to ethical guidelines. Ultimately, LLMs should be viewed as complementary tools within dental education, with careful consideration of their limitations."
W4414532985,,10.1200/cci-25-00076,2025,Mitigating Ethical Issues for Large Language Models in Oncology: A Systematic Review,Shuang Zhou,9,JCO Clinical Cancer Informatics,4,"PURPOSE: Large language models (LLMs) have demonstrated remarkable versatility in oncology applications, such as cancer staging and survival analysis. Despite their potential, ethical concerns such as data privacy breaches, bias in training data, lack of transparency, and risks associated with erroneous outputs pose significant challenges to their adoption in high-stakes oncology settings. Therefore, we aim to explore the ethical challenges associated with LLM-based applications in oncology and evaluate emerging techniques designed to address these issues. METHODS: Following the Preferred Reporting Items for Systematic Reviews and Meta-Analyses framework, a systematic review was conducted to evaluate publications related to ethical issues of LLMs in oncology across eight academic databases (eg, PubMed, Web of Science, and Embase) between January 1, 2019, and December 31, 2024. RESULTS: The search retrieved 4,319 published articles, of which 65 publications were preserved and included in our analysis. We identified six prevalent ethical challenges in oncology, including trust, equity, privacy, transparency, nonmaleficence, and accountability. We then evaluated emerging technical solutions to mitigate ethical challenges and summarized evaluation metrics used to assess these solutions' effectiveness. CONCLUSION: This review provides actionable recommendations for responsibly deploying LLMs in oncology, ensuring adherence to ethical guidelines, and fostering improved patient outcomes. By bridging technical and clinical perspectives, this review offers a foundational framework for advancing ethical artificial intelligence applications in oncology and highlights areas for future research."
W4411407939,,10.34133/hds.0322,2025,LitAutoScreener: Development and Validation of an Automated Literature Screening Tool in Evidence-Based Medicine Driven by Large Language Models,Yiming Tao,6,Health Data Science,9,"Background: The traditional manual literature screening approach is limited by its time-consuming nature and high labor costs. A pressing issue is how to leverage large language models to enhance the efficiency and quality of evidence-based evaluations of drug efficacy and safety. Methods: This study utilized a manually curated reference literature database—comprising vaccine, hypoglycemic agent, and antidepressant evaluation studies—previously developed by our team through conventional systematic review methods. This validated database served as the gold standard for the development and optimization of LitAutoScreener. Following the PICOS (Population, Intervention, Comparison, Outcomes, Study Design) principles, a chain-of-thought reasoning approach with few-shot learning prompts was implemented to develop the screening algorithm. We subsequently evaluated the performance of LitAutoScreener using 2 independent validation cohorts, assessing both classification accuracy and processing efficiency. Results: For respiratory syncytial virus vaccine safety validation title–abstract screening, our tools based on GPT (GPT-4o), Kimi (moonshot-v1-128k), and DeepSeek (deepseek-chat 2.5) demonstrated high accuracy in inclusion/exclusion decisions (99.38%, 98.94%, and 98.85%, respectively). Recall rates were 100.00%, 99.13%, and 98.26%, with statistically significant performance differences ( χ 2 = 5.99, P = 0.048), where GPT outperformed the other models. Exclusion reason concordance rates were 98.85%, 94.79%, and 96.47% ( χ 2 = 30.22, P < 0.001). In full-text screening, all models maintained perfect recall (100.00%), with accuracies of 100.00% (GPT), 100.00% (Kimi), and 99.45% (DeepSeek). Processing times averaged 1 to 5 s per article for title–abstract screening and 60 s for full-text processing (including PDF preprocessing). Conclusions: LitAutoScreener offers a new approach for efficient literature screening in drug intervention studies, achieving high accuracy and significantly improving screening efficiency."
W7151398448,,10.47626/2237-6089-2025-1237,2026,The Role of Large Language Models in Teaching Psychiatric Semiology: A Systematic Review,Vinícius Vicente Soares,5,Trends in Psychiatry and Psychotherapy,0,"INTRODUCTION: Teaching psychiatric semiology faces challenges such as the limited availability of real patients for educational purposes and a shortage of specialized instructors. This systematic review investigated the applications of Large Language Models (LLMs), exemplified by ChatGPT, to address these gaps. METHODS: Studies published between November 2022 and September 17, 2025, were identified through searches in PubMed/MEDLINE, Scopus, Web of Science, IEEE Xplore, EBSCO, BVS, and EMBASE databases. Two independent reviewers conducted the systematic review following the PRISMA 2020 checklist, and the methodological quality of the included studies was individually assessed using the Mixed Methods Appraisal Tool (MMAT). RESULTS: Of the 1,549 studies initially identified, two met the inclusion criteria, involving medical students and educators in Canada and Switzerland. LLMs demonstrated performance comparable to human experts in creating diagnostic tests and clinical vignettes but exhibited occasional simplifications and algorithmic biases. Participants reported positive perceptions of the tools' efficiency and practicality, but emphasized the need for specialized supervision and attention to potential privacy issues and superficiality in complex cases. CONCLUSIONS: LLMs show potential as valuable supplementary resources for teaching psychiatric semiology, especially in settings with shortages of teachers and available patients for training. However, their limitations, including cultural and algorithmic biases, potential privacy risks, and superficial content generation, require close monitoring by experienced professionals. Although current evidence is preliminary, prospects are promising, highlighting the need for more robust and long-term studies to evaluate the educational use of LLMs."
W4320342216,10.48550/arxiv.2302.03495,10.48550/arxiv.2302.03495,2023,Can ChatGPT Write a Good Boolean Query for Systematic Review Literature Search?,Shuai Wang,4,arXiv (Cornell University),40,"Systematic reviews are comprehensive reviews of the literature for a highly focused research question. These reviews are often treated as the highest form of evidence in evidence-based medicine, and are the key strategy to answer research questions in the medical field. To create a high-quality systematic review, complex Boolean queries are often constructed to retrieve studies for the review topic. However, it often takes a long time for systematic review researchers to construct a high quality systematic review Boolean query, and often the resulting queries are far from effective. Poor queries may lead to biased or invalid reviews, because they missed to retrieve key evidence, or to extensive increase in review costs, because they retrieved too many irrelevant studies. Recent advances in Transformer-based generative models have shown great potential to effectively follow instructions from users and generate answers based on the instructions being made. In this paper, we investigate the effectiveness of the latest of such models, ChatGPT, in generating effective Boolean queries for systematic review literature search. Through a number of extensive experiments on standard test collections for the task, we find that ChatGPT is capable of generating queries that lead to high search precision, although trading-off this for recall. Overall, our study demonstrates the potential of ChatGPT in generating effective Boolean queries for systematic review literature search. The ability of ChatGPT to follow complex instructions and generate queries with high precision makes it a valuable tool for researchers conducting systematic reviews, particularly for rapid reviews where time is a constraint and often trading-off higher precision for lower recall is acceptable."
W7143806077,,10.3389/fmed.2026.1810098,2026,Large language models for primary care ophthalmic education: a systematic review,Shuang Li,7,Frontiers in Medicine,2,"Background: Primary care physicians (PCPs) are a critical first contact for eye-health screening, risk stratification, and referral, yet ophthalmology training and point-of-care support in primary care remain insufficient. Recent advances in generative artificial intelligence (generative AI), particularly large language models (LLMs), may help address these gaps through conversational, scenario-based learning and structured feedback. However, the educational effectiveness, reproducibility, and safety boundaries of LLM-enabled tools in primary care ophthalmology remain unclear. Methods: We conducted a systematic review of studies evaluating or applying LLMs in ophthalmic education, training, assessment, or primary care-relevant clinical support. PubMed, Web of Science Core Collection, and Scopus were searched from January 1, 2020 to December 31, 2025 using combined terms related to LLMs/generative AI, ophthalmology, and education or assessment. Citation chaining was also performed to reduce omission. Two reviewers independently screened records and extracted data. Results: The evidence base is dominated by vignette-based benchmarks, comparative scoring studies, and evaluations conducted in limited-sample or controlled settings; prospective real-world validation using learner transfer, clinical behavior change, workflow impact, or patient outcomes remains scarce. Across studies, LLMs can serve as ""cognitive apprenticeship"" partners by externalizing clinical reasoning and enabling repeated practice in key-feature extraction, differential diagnosis, risk stratification, and referral-threshold decisions. Applications include triage/reasoning drills, virtual patient interviewing, and support for structured referrals, documentation, and patient education, often strengthened by retrieval-augmented generation. Most studies benchmarked outputs against expert consensus or guidelines, but scoring rubrics and reference standards varied widely, limiting cross-study comparability. Some reports noted that adding clinical photographs could reduce accuracy, suggesting current multimodal models are better suited for history-based reasoning than fine-grained image interpretation. Limitations include heterogeneity, rapid model iteration, reproducibility challenges, multimodal instability, and safety risks such as hallucination, bias, and automation bias. Commonly recommended safeguards include retrieval grounding, source attribution, red-flag checklists, and human-in-the-loop review. Conclusion: With clearly defined task scopes and robust safeguards, LLMs may improve the accessibility and efficiency of primary care ophthalmic education, but should augment rather than replace expert judgment. Future work should prioritize pragmatic multicenter trials, mixed-method implementation studies, and standardized cross-lingual evaluations to define safe and effective implementation pathways."
W7129095804,,10.17605/osf.io/rzju8,2026,Performance of Large Language Models in Title and Abstract Screening of Preclinical Systematic Reviews - Study Registration,Francisco Marques da Silva Neto,4,,0,"This preregistered methodological validation study evaluates how well large language models perform title/abstract screening for preclinical systematic reviews from BRISA. We will compare LLM decisions against the reviews’ final full-text inclusion (gold standard), estimating sensitivity/specificity, efficiency gains, and test–retest reproducibility across multiple datasets."
W7169782079,,10.1108/jstpm-01-2026-0015,2026,Hallucinations in large language models within academic contexts: a systematic review,Ke En Lai,4,Journal of Science and Technology Policy Management,1,"Purpose This study aims to examine the phenomenon of hallucinations in large language models (LLMs) within academic contexts, focusing on their manifestations, causes and implications for academic integrity, research quality and responsible artificial intelligence adoption in higher education. Design/methodology/approach A systematic literature review was conducted in accordance with PRISMA 2020 guidelines. Searches across Scopus, Web of Science and Emerald Insight databases using keywords related to AI hallucination and academic applications, of which 25 peer-reviewed journal articles met the inclusion criteria. Qualitative thematic analysis was performed using NVivo 14 to synthesise evidence on hallucination types, academic applications, impacts and mitigation strategies. Findings Six recurring types of hallucinations were identified, with fabricated or inaccurate citations emerging as the most prevalent. The findings indicate that hallucinations systematically compromise academic writing quality, distort assessment processes and undermine epistemic trust in scholarly outputs. Variation in hallucination rates across models and disciplines highlights their context-dependent nature. Key contributing factors include probabilistic text generation, limitations in training data, insufficient contextual understanding and the absence of robust verification mechanisms. Practical implications It further contributes a structured classification of hallucination types and a multi-layered governance approach to inform institutional policy and responsible AI adoption. Social implications Addressing hallucinations in academic knowledge production is essential for preserving public trust in higher education and safeguarding the societal value of scholarly research. Originality/value This study advances existing knowledge by developing an integrated conceptual perspective linking hallucinations to epistemic risk, information integrity and digital trust."
W7127186092,,10.1007/s10586-025-05891-z,2026,"The rise of hallucination in large language models: systematic reviews, performance analysis and challenges",Shamsu Abdullahi,3,Cluster Computing,5,
W7155029538,,10.1145/3842759,2026,LLM4Log: A Systematic Review of Large Language Model-based Log Analysis,Zeyang Ma,3,ACM Transactions on Software Engineering and Methodology,1,"Software systems generate massive, evolving, semi-structured logs that are central to reliability engineering and AIOps, yet difficult to analyze at scale under drift and limited labels. Recent advances in pretrained Transformer models and instructiontuned large language models (LLMs) have reshaped log analysis by enabling semantic generalization and cross-source evidence integration, but also introducing deployment risks such as context limits, latency and cost, privacy constraints, and hallucinations. This paper presents LLM4Log , a systematic review of LLM-based log analysis across the end-to-end pipeline, from upstream logging-statement generation and maintenance to log parsing/structuring and downstream tasks including anomaly detection, failure prediction, root cause analysis, and log summarization. Following a structured search and manual screening protocol, we completed literature collection in November 2025 and identified 145 unique papers across seven logging tasks. We organize the research area through a unified, task-driven taxonomy, summarize common design patterns (prompting/ICL, retrieval grounding, fine-tuning, tool/agent augmentation, and verification), and analyze evaluation practices, datasets, metrics, and reproducibility. Based on these cross-paper analyses, we summarize key lessons and open challenges for reliable real-world adoption. We emphasize robustness under drift and long-tail events, grounding and faithfulness for operator-facing outputs, and deployment-oriented designs with verifiable behavior."
W7113238458,,,2025,The Social Science of Large Language Models: A Systematic Review and Computational Taxonomy,Xiao Jia,1,OSF Preprints (OSF Preprints),0,"This project preregisters a systematic review protocol on the social science of large language models (LLMs). LLMs such as GPT, Claude, and LLaMA are increasingly used in contexts that intersect with human social behavior, yet existing studies are scattered across disciplines. We will systematically search Web of Science, Scopus, ACM Digital Library, IEEE Xplore, PubMed, arXiv, Consensus, and Semantic Scholar, supplemented with hand-searching of top journals and conferences. Inclusion criteria focus on studies that directly involve LLMs and social science constructs; exclusion criteria filter out purely engineering applications and studies where LLMs are only used as tools. Screening will follow PRISMA 2020 guidelines, with two reviewers conducting independent title/abstract and full-text reviews. The expected outcome is a comprehensive overview of ~150 core studies, organized into a conceptual framework that highlights LLMs’ capacities and limitations. To ensure systematic categorization and to handle the large body of literature, we will employ an unsupervised machine learning approach to generate a computational taxonomy of the studies. This will involve clustering studies based on content similarity using sentence embeddings and dimensionality reduction techniques like UMAP. The resulting taxonomy will reveal the latent structure of the field and identify key research themes across various social science domains. This review will serve as a resource for researchers in AI, psychology, sociology, political science, and economics, and will provide methodological guidance for future work. Data extraction templates and coding sheets will be shared openly to enhance transparency and reproducibility."
W4416947049,,10.2196/84120,2025,Knowledge-Practice Performance Gap in Clinical Large Language Models: Systematic Review of 39 Benchmarks,Eun Jeong Gong,4,Journal of Medical Internet Research,41,"BACKGROUND: The evaluation of large language models (LLMs) in medicine has undergone a shift from knowledge-based testing to practice-based assessment, representing an evolution in how we measure artificial intelligence readiness for clinical deployment. While LLMs now routinely exceed human performance on medical licensing examinations, their translation to clinical practice remains poorly characterized. OBJECTIVE: This systematic review aims to categorize and analyze medical LLM benchmarks, examining performance patterns across different evaluation paradigms and identifying gaps in current assessment methodologies. METHODS: The protocol was registered at PROSPERO (CRD420251139729). Four databases (MEDLINE/PubMed, Embase/Ovid, Cochrane Library, and arXiv) were searched from inception to August 31, 2025, using keywords related to clinical medicine benchmarks in LLMs. Studies were included if they (1) investigated clinical medicine benchmarks in LLMs, (2) were published in English, and (3) were available in full-text. Studies were excluded if they evaluated nonmedical domains or lacked benchmark validation. Methodological quality was assessed using the Mixed Methods Appraisal Tool (version 2018) by 2 independent reviewers (κ=0.91). Due to heterogeneity in evaluation metrics preventing meta-analysis, narrative synthesis was conducted using structured categorization of benchmark types. RESULTS: From 3917 screened records, 39 medical LLM benchmarks were identified and categorized into 21 (54%) knowledge-based, 15 (38%) practice-based, and 3 (8%) hybrid frameworks. These benchmarks collectively encompass over 2.3 million questions across 45 languages and 172 medical specialties. Traditional knowledge-based benchmarks show saturation with leading models achieving 84%-90% accuracy on USMLE (United States Medical Licensing Examination)-style examinations, approaching or exceeding average physician performance. However, practice-based assessments reveal performance challenges, with specific benchmarks showing varied results: DiagnosisArena 45.82% (95% CI 42.9%-48.8%), MedAgentBench 69.67% (95% CI 64.2%-74.6%), and HealthBench 60% (95% CI 58.6%-61.3%) success rates, with practice-based benchmarks showing lower performance (45%-69%) compared to knowledge benchmarks (84%-90%). Task-specific analysis revealed differential performance patterns: factual retrieval maintained 85%-93% accuracy, clinical reasoning dropped to 50%-60%, diagnostic tasks achieved 45%-55% success, and safety assessment showed significant gaps at 40%-50% accuracy despite being life-critical. Geographic representation spans 6 continents with 18 (46%) benchmarks, incorporating non-English content. Quality assessment revealed 26% (10/39) of benchmarks had insufficient methodological reporting for complete evaluation. CONCLUSIONS: This systematic review provides the first comprehensive analysis quantifying the significant ""knowledge-practice gap"" in medical artificial intelligence: high performance on knowledge-based examinations (84%-90%) does not translate to clinical competence (45%-69%), with safety assessments at 40%-50%. Our findings provide quantitative evidence for regulators and health systems that examination scores are insufficient and misleading proxies for clinical readiness. This review concludes that autonomous deployment is not currently justifiable and that all evidence-based implementation strategies must mandate practice-oriented validation and robust human-in-the-loop oversight to ensure patient safety. TRIAL REGISTRATION: PROSPERO CRD420251139729; https://www.crd.york.ac.uk/PROSPERO/view/CRD420251139729."
W7223172290,,10.1038/s41746-026-03349-z,2026,Large language models for multimodal clinical predictive modeling: a systematic review,Levi Kaster,4,npj Digital Medicine,0,"The electronic health record (EHR) comprises structured and unstructured data, both of which are integral for comprehensively representing patient history. The emergence of large language models (LLMs) has enabled the incorporation of unstructured data modalities into machine learning models that leverage EHR data. In this systematic review, we characterize studies that utilize LLM-enabled multimodal predictive modeling with EHR data. Following a screening of 641 studies, 57 met inclusion criteria. We identified three strategies for LLM-based multimodal integration: embedding, extraction, and text fusion. Delineated by the format of the unstructured data at the time of fusion, all strategies yielded performance gains (average 6.4% increase) over single modality models. LLMs facilitated modeling by generating note-embeddings for downstream prediction, extracting clinically informative variables, and summarizing long text documents prior to integration. Building on our review, we identified LLM-enhanced predictive modeling considerations that include sample size, interpretability, and computational costs. This study supports the potential value of LLMs for EHR-based predictive modeling and provides a framework for future multimodal modeling tools."
W4411253453,,10.1101/2025.06.10.25329350,2025,Large Language Models for Sentiment Analysis in Healthcare: A Systematic Review Protocol,Ravi Shankar,3,medRxiv,0,"Abstract Large language models (LLMs) have emerged as powerful tools for sentiment analysis in healthcare, offering potential advantages in capturing contextual information and semantic relationships in complex medical text. Healthcare sentiment analysis presents unique challenges due to domain-specific terminology, privacy regulations, and the nuanced nature of patient experiences. This systematic review protocol outlines a comprehensive methodology to investigate the application of LLMs for sentiment analysis across healthcare settings, including analysis of patient feedback, social media content, and electronic health records. By synthesizing current evidence, we aim to provide insights for researchers, clinicians, and policymakers on the effectiveness, limitations, and ethical considerations of these advanced natural language processing techniques. We will conduct a systematic review following PRISMA-P 2015 guidelines and using the PICOS framework. The search strategy will encompass eight major databases (PubMed, Web of Science, Embase, CINAHL, MEDLINE, The Cochrane Library, PsycINFO, and Scopus) using a comprehensive search string combining terms related to LLMs, sentiment analysis, and healthcare contexts. We will include peer-reviewed studies published between 2018 (corresponding to BERT’s introduction) and March 2025 that focus on LLM applications for healthcare sentiment analysis with reported performance metrics or qualitative evaluations. Two independent reviewers will screen titles/abstracts and full texts, with disagreements resolved through discussion or third-reviewer consultation. Data extraction will capture study characteristics, research objectives, dataset details, LLM architecture specifications, fine-tuning approaches, performance metrics, and implementation challenges. Quality assessment will employ a modified QUADAS-2 tool and the Cochrane Risk of Bias tool. We will conduct narrative synthesis of the findings, organizing them thematically according to our research questions, with meta-analysis performed if study heterogeneity permits. PROSPERO registration number CRD420251012298 Strengths and limitations of this study This is the first systematic review to comprehensively examine large language models for sentiment analysis specifically within healthcare contexts, addressing a significant gap in the literature. The review’s rigorous methodology follows PRISMA-P guidelines and employs dual independent screening, data extraction, and quality assessment to ensure thoroughness and minimize bias. The inclusion of diverse healthcare text sources (patient feedback, social media, electronic health records) allows for a comprehensive understanding of LLM applications across the healthcare information ecosystem. By focusing on studies published since 2018 (when BERT was introduced), the review captures the most relevant technological developments while excluding outdated approaches. A limitation of this study is the expected heterogeneity across included studies (varying LLM architectures, datasets, metrics, and implementation contexts), which may preclude meaningful meta-analysis and limit definitive conclusions about relative performance, resulting in more descriptive than prescriptive findings."
W7166728658,,10.1192/bjo.2026.11056,2026,Bias Evaluation in Medical Applications of Large Language Models: A Systematic Review,Priya Gupta,5,BJPsych Open,0,"Aims: Large language models (LLMs) are increasingly evaluated and deployed in healthcare, yet unrecognised bias risks reinforcing inequities. This risk is particularly salient in psychiatry, where biased outputs could influence risk formulation, diagnostic reasoning, triage, and therapeutic communication. We examined whether and how bias is evaluated in studies of medical applications of LLMs, and summarised methodologies, reporting practices,and limitations of current bias detection and mitigation approaches to support safe and trustworthy implementation. Methods: We searched databases (EMBASE, MEDLINE, PsycINFO, PubMed, ACL Anthology, ACM Digital Library, arXiv, medRxiv, and bioRxiv; 2017–June 2025), following PRISMA guidance. After de-duplication, >1,800 records were retrieved for title/abstract screening. We assessed 3,664 full-text articles against pre-specified eligibility criteria. Studies were included if they evaluated a medical application of an LLM and reported any explicit bias assessment. Studies that otherwise met eligibility for medical LLM evaluation but did not assess bias were recorded separately to quantify the evidence gap. Screening and data extraction were conducted by multiple reviewers with disagreements resolved by discussion. We extracted specialty (including psychiatry), use case (e.g. decision support, documentation, patient communication), evaluation setting (benchmark/vignette vs clinical data), protected attributes and bias targets assessed, bias methods/metrics, and mitigation strategies proposed or tested. Findings were synthesised narratively with descriptive counts across specialties and use cases. Results: Of 3,664 full texts screened, 278 studies met inclusion by explicitly assessing bias. A further 757 studies evaluated medical applications of LLMs and otherwise met eligibility criteria but did not report any bias assessment, indicating that most medical LLM evaluations omit bias evaluation. Included studies spanned clinical decision support, medical documentation, patient communication, education, and biomedical research across multiple specialties. Bias assessment rates varied by specialty: psychiatry 21/180 (11.7%), radiology 7/107 (6.5%), and oncology 5/51 (9.8%). Bias assessments most commonly examined demographic bias (gender, race/ethnicity, age) and variation in symptom or diseasepresentation. Methodologies included prompt-based probing (e.g. clinical vignettes), counterfactual testing, and red teaming to elicit biased responses; statistical comparisons of performance across groups (including fairness-adjusted performance metrics such as the FAP score); and clinician-led safety evaluation frameworks assessing output suitability. Proposed mitigation strategies included more diverse training data, reinforcement learning with human feedback (RLHF), direct preference optimisation, and ongoing human oversight. Conclusion: Despite rapid growth in medical LLM research, explicit bias assessment remains inconsistent and frequently absent. Standardised, transparent bias evaluation across patient populations, tasks, and clinical contexts is needed to support safe implementation and ensure equitable provision."
W4362522969,,10.4103/singaporemedj.smj-2023-055,2023,The rise of artificial intelligence: addressing the impact of large language models such as ChatGPT on scientific publications,Tiing Leong Ang,4,Singapore Medical Journal,31,"INTRODUCTION Artificial intelligence (AI) permeates our modern society. Chatbots are being used by commercial companies as an interface with their customers, and Internet search engines rely heavily on AI to function. Smart homes, navigation systems and immigration clearance using facial recognition are other examples of the use of AI in our daily lives. In general, AI is an essential element for biotechnology. In biotechnology and research, the applications of AI include drug target identification, image screening, predictive modelling of disease states and outcomes, assessment of functional and structural genomics, metabolomics and proteomics[1] among others. In scientific writing, AI is used in literature search in biomedical databases such as PubMed, and it is present in a rudimentary form in word processing software that researchers use to write scientific manuscripts. AI can be broadly classified into narrow AI, general AI and artificial superintelligence (ASI). Narrow AI is created to solve specific problems. General AI aims to create machines capable of reasoning and thinking like humans. ASI would be capable of outperforming humans. Currently available AI systems are considered narrow AI. LAUNCH OF ChatGPT We entered uncharted territories in scientific writing with the launch of Chat Generative Pre-trained Transformer (ChatGPT; available from: https://chat.openai.com/chat) on 30 November 2022 by OpenAI, an AI research and deployment company. ChatGPT is a large language model (LLM) that has the ability to generate human-like text in a conversational way, including complete paragraphs or even essays, based on specific prompts and context, by mimicking the statistical patterns of language from a huge database of text collated from the Internet. It can potentially write an entire scientific manuscript. Using ChatGPT creates the illusion of conveniently having an expert readily at hand to answer one’s questions at all times. The current version of ChatGPT is a prototype that was trained on dataset that is updated up to 2021. It is not connected to the Internet for real-time updates and does not have access to new knowledge derived after 2021. Therefore, it is to be expected that there will be data gaps in ChatGPT, in terms of new information that becomes available after 2021. In future, the currency of knowledge may be less of an issue if there is continuous input of new data for software training and development, similar to how academics and professionals continuously acquire new knowledge and update themselves. The capabilities of LLMs are limited to the textual training data used, and this can result in problems of reliability and bias. In addition, each LLM has a certain amount of memory, and hence can accept only a finite amount of input to form the context for output generation. There are several issues that need to be properly addressed even at this nascent phase of its development.[2] Concerns have been raised about its role in scientific writing, including whether it can be considered as a co-author, its potential negative impact, such as whether it will result in factual inaccuracies, plagiarism or even outright scientific fraud, and whether it facilitates or blunts the academic development of an individual. Issue of authorship and transparency ChatGPT has been used to draft scientific abstracts and papers, and in the process, it has even been listed as a co-author in academic publications.[3,4,5] The International Committee of Medical Journal Editors (ICMJE) recommends that authorship be based on meeting the following four criteria: (a) substantial contributions to the conception or design of the work, or the acquisition, analysis or interpretation of data for the work; (b) drafting the work or revising it critically for important intellectual content; (c) final approval of the version to be published; and (d) agreement to be accountable for all aspects of the work in ensuring that questions related to the accuracy or integrity of any part of the work are appropriately investigated and resolved.[6] Thus, it is clear that even if ChatGPT or more sophisticated AI programs were used to write the entire manuscript in future, they would still fall short of the eligibility criteria, especially with respect to the issue of personal accountability by authors. AI tools cannot assume such a responsibility. There is consensus by many researchers and journals that AI models cannot be credited as authors,[7,8] and this is the position that the Singapore Medical Journal holds. Transparency and integrity are crucial elements in scientific writing. A submitted academic manuscript must be the original work of the authors, and the wholesale use of ChatGPT without formal acknowledgement is akin to plagiarism. If ChatGPT or any other LLM is to be used as part of the manuscript preparation, there needs to be clarity on how it is being used when the manuscript is written, and whether and how data derived from ChatGPT and other LLMs are further verified. This must be appropriately documented either in the methods or the acknowledgement section of the manuscript. Such transparency is important to prevent erosion of trust when the manuscript is being appraised. It has also been advocated that as an additional safeguard, AI output detectors be incorporated into the editorial process to alert journal editors and reviewers — similar in concept to current approaches of using AI algorithms to screen for plagiarism by checking the similarity index — and to check for bot-like behaviour in the Internet. Tackling the issues of inaccuracies and quality There are concerns about data gaps, factual inaccuracies and bias when ChatGPT is used by authors to draft manuscripts. Although the text written by ChatGPT may appear credible, it has been shown that the generated information can be pure confabulation, with a combination of both facts and fabricated information, or totally fictitious pseudoscientific material. This has been described as artificial hallucination.[9] In future, continuous or regularly scheduled updates of the AI software may help to attenuate this problem of knowledge gaps in the software. Even if there is regular training and updating of the AI software, there may still be a lag phase. Bias in scientific interpretation and writing may also exist, and this could be related to the training of the AI algorithm. It has been suggested that the way the ChatGPT was trained created a left-wing, liberal bias in its responses.[10] The ease of accessing LLMs may result in erosion of skills and expertise needed for high-quality academic publication and promote proliferation of low-quality scientific papers. Therefore, concerns about integrity, accuracy and deterioration of quality with the use of ChatGPT and similar programs in academic writing are justified. However, problems such as academic bias, inaccuracies, ghost authorship, plagiarism and fraud may still arise even in the absence of AI tools like ChatGPT. It is thus crucial that the concept of research integrity be continuously emphasised to all researchers, journals screen and sieve out submissions of dubious quality, and academics and reviewers develop and retain the skillset for critical appraisal, discernment of information and source verification. Clinicians and scientists who read scientific publications must be able to think critically and not accept information at face value. We have already pivoted away from eminence-based medicine to evidence-based medicine. Human verification of information generated by AI remains important. In reality, critical thinking is important not only within academia, but also generally in life, and one needs to be able to separate facts from myths, half-truths, disinformation, pure propaganda and lies. Having insights on how an AI program was developed would allow one to have a better appreciation of its limitation and the potential for bias and errors. A case in point is the knowledge that the current version of ChatGPT has not been trained on new information derived after 2021, and researchers would thus know that ChatGPT cannot be a trusted source for information after 2021. However, a full understanding of the developmental process of the AI model, including the type and extent of information used in its training, and the potential personal bias of people involved in programming may not always be possible, with such systems existing as a black box due to proprietary reasons.[11] Human verification of the accuracy of AI outputs is crucial, and AI should be considered an additional resource to obtain data, with potential for gaps or even inaccuracies, and not a purveyor of absolute truth. CONCEPT OF COLLABORATIVE INTELLIGENCE ChatGPT or other LLMs can facilitate the conduct of research by compiling pre-existing information. For instance, it can be used to draft a study protocol or format the manuscript after completion of the study, which speeds up the writing process. AI can also help improve the language of non-native English speakers and allow translation of an English manuscript to other languages, thus improving access to non-English readers and broadening the reach of journals. However, AI cannot generate new knowledge out of a vacuum when the research has yet to be conducted. Well-trained researchers with deep domain expertise are the ones at the forefront to conduct novel research, generate new findings and achieve scientific breakthroughs. The human interface remains essential for scientific progress. LLMs can be writing assistants and journal assistants but cannot replace human authors or human editors. At the same time, continued training of LLMs will and should occur, so that they can be further improved to help the writing and publishing process. We need to optimise collaborative intelligence, where humans and AI work together, to ensure transparency, integrity and scientific rigour. CONCLUSION AI can be a boon or a bane. While AI is an enabler and a game changer, there exist privacy and ethical concerns, possibility of bias, potential legal liabilities and concerns about validity.[12] In the novel ‘Origin’ written by Dan Brown, a terminally ill Edmond Kirsch instructed Winston, the AI personal assistant he invented, to maximise viewership and hype up an important announcement he was going to make on the Internet. Winston then orchestrated Kirsch’s death during his presentation to make him a martyr, thus boosting online viewership from 3 million to more than 200 million people. AI can facilitate progress, and hence, we should continue to embrace AI and establish collaborative intelligence, especially as it continues to evolve, moving from narrow AI to higher levels of function, such as general AI, or even ASI in future. However, it is important to set up guardrails and a framework to guide its application, so that the potential of AI can be harnessed appropriately."
W4385848697,10.48550/arxiv.2308.06610,10.48550/arxiv.2308.06610,2023,Bio-SIEVE: Exploring Instruction Tuning Large Language Models for Systematic Review Automation,Ambrose Robinson,7,arXiv (Cornell University),9,"Medical systematic reviews can be very costly and resource intensive. We explore how Large Language Models (LLMs) can support and be trained to perform literature screening when provided with a detailed set of selection criteria. Specifically, we instruction tune LLaMA and Guanaco models to perform abstract screening for medical systematic reviews. Our best model, Bio-SIEVE, outperforms both ChatGPT and trained traditional approaches, and generalises better across medical domains. However, there remains the challenge of adapting the model to safety-first scenarios. We also explore the impact of multi-task training with Bio-SIEVE-Multi, including tasks such as PICO extraction and exclusion reasoning, but find that it is unable to match single-task Bio-SIEVE's performance. We see Bio-SIEVE as an important step towards specialising LLMs for the biomedical systematic review process and explore its future developmental opportunities. We release our models, code and a list of DOIs to reconstruct our dataset for reproducibility."
W7127197632,,10.1007/978-3-032-15404-0_5,2026,The Impact of Large Language Models in Healthcare: A Systematic Review of AI Applications,Abhishek Kumar Maurya,3,Lecture notes in networks and systems,0,
W4405750564,,10.1016/j.jval.2024.10.2293,2024,MSR59 Evaluating Systematic Literature Review Screening Performance of Seven Large Language Models in Response to Different Prompting Strategies,Ghayath Janoudi,4,Value in Health,1,
W7104463143,,10.5281/zenodo.17561086,2025,"Large Language Models and ChatGPT in Medical Sciences: Foundations, Capabilities, and Challenges",Amir Tahavvori,27,Zenodo (CERN European Organization for Nuclear Research),0,"Large Language Models (LLMs) such as ChatGPT have transformed how medical knowledge is accessed and used. Built on transformer-based deep learning architectures, these systems are trained on massive datasets containing biomedical and general language sources. This training enables them to interpret clinical text, summarize research findings, and generate coherent, contextually relevant responses. In medicine, LLMs now support a wide range of applications including literature screening, clinical documentation, education, and patient interaction. ChatGPT, in particular, can help synthesize evidence, draft research manuscripts, explain complex medical concepts, and assist in decision support when appropriately guided. These tools have accelerated many aspects of medical research and practice by improving efficiency and accessibility of information. Despite these advances, challenges remain significant. LLMs can produce inaccurate or fabricated information, known as hallucinations, which poses risks in clinical and research settings. Their training data may contain biases that influence outputs, and their decision-making processes are often opaque. Ethical concerns also arise regarding authorship, accountability, and data privacy. To integrate LLMs responsibly, transparency about their use and human oversight are essential. When applied thoughtfully, these tools can enhance scientific discovery, education, and healthcare delivery while maintaining the rigor and ethical standards that underpin medical science."
W7161385519,,10.38124/ijisrt/26apr2488,2026,Autonomous KGs for Large Language Model Agents in Industry: A Systematic Review,Asma Hamad Alharbi,2,International Journal of Innovative Science and Research Technology (IJISRT),0,"The integration of large language models (LLMs) with knowledge graphs (KGs) has emerged as a promising approach to enhance reasoning, explainability, and decision-making in intelligent systems. However, LLMs inherently lack structured knowledge representation, leading to limitations such as hallucinations and reduced reliability, particularly in industrial applications. This study presents a systematic review of autonomous knowledge graphs for LLM-based agents, focusing on their role in enabling knowledge-driven and agentic AI systems in industry. Guided by PRISMA methodology, a comprehensive search across major scientific databases resulted in 33 relevant studies, which were classified into four research dimensions: architectures, KG construction and evolution, KG-based reasoning, and industrial applications. The findings reveal that multi-agent architectures and GraphRAG-based approaches dominate current research, while LLMs enable automated and dynamic knowledge graph lifecycle management. Furthermore, applications in manufacturing, digital twins, and industrial IoT demonstrate significant improvements in efficiency and decision-making. Despite these advancements, challenges related to scalability, computational cost, domain dependency, and lack of real-world validation persist. This review concludes that autonomous knowledge graphs play a critical role in advancing LLM-based agents toward reliable and scalable industrial deployment, while highlighting key research gaps and future directions for developing robust, explainable, and industry-ready AI systems."
W4414078895,,10.1016/j.cont.2025.102246,2025,322 - Role of Large Language Models in Urology: A systematic review and meta-analysis,Mohamed Javid Raja Iyub,11,Continence,1,"Hypothesis / aims of study Large Language Models (LLMs) have demonstrated transformative potential, with promising insights that could potentially enhance clinical practice. Their application in overall healthcare practices is increasingly noted, but their cumulative usage in the field of urology is less explored. Our study aimed to systematically review the current evidence on the usage of LLMs in urology and perform a quantitative analysis of comparable studies. Study design, materials and methods We searched electronic databases namely, Pubmed, Embase, Web of Science, and Scopus to include studies involving urological data or patients in which LLMs were utilized in clinical management or other relevant applications. The studies were grouped based on the application of the LLM that was being studied as: 1) Answering FAQs, 2) Patient Information Materials, 3) Clinical Practice, 4) Medical examination and 5) Other applications. The quality assessment was done utilising the Newcastle-Ottawa Scale. Results A total of 814 articles were found, and after the removal of duplicates and screening of the articles, 39 studies were included in our review. Various applications of LLMs in different domains were listed, and details regarding the training, outcomes, and limitations were studied. ChatGPT was the most commonly studied LLM. The pooled accuracy of the output of various LLMs on various medical examinations was found to be 63.24 (CI 53.69 – 72.72), and a forest plot was constructed. Studies analysing the output of LLMs in answering frequently asked questions reported an accuracy of 67.08% to 100%, with ChatGPT – 4 outperforming ChatGPT – 3.5. Interpretation of results The findings of this systematic review and meta-analysis underscore the growing role of LLMs in urology, highlighting their potential to enhance clinical practice across diverse applications. The pooled accuracy of 63.24% for LLM performance on medical examinations reflects moderate reliability, with variability likely influenced by differences in training data, model architecture, and evaluation methods. Notably, ChatGPT emerged as the most frequently studied LLM, with its latest version (ChatGPT-4) demonstrating superior accuracy (67.08% to 100%) in answering frequently asked questions compared to its predecessor, ChatGPT-3.5. This suggests that advancements in LLM iterations can significantly improve performance, particularly in patient-facing tasks. However, the results also reveal a lack of standardization in evaluation metrics and methodologies across studies, which poses challenges for consistent benchmarking and integration into clinical workflows. Concluding message LLMs' applications are diverse and have augmented urological practice. However, uniform evaluation methods and performance metrics for assessing the output generated for various purposes are needed to further streamline the use and synchronous incorporation of LLM in regular urological practice. Download: Download high-res image (104KB) Download: Download full-size image Figure 1 . Funding None Clinical Trial No Subjects None"
W4415071193,,10.7759/cureus.94300,2025,Performance of ChatGPT and Large Language Models on Medical Licensing Exams Worldwide: A Systematic Review and Network Meta-Analysis With Meta-Regression,Alousious Kasagga,9,Cureus,10,"Large language models (LLMs) are increasingly being tested on national medical licensing examinations, yet existing research is fragmented across models, exam systems, and languages. This study is the first meta-analysis to systematically assess LLM performance across multiple medical licensing exams and languages using pooled estimates, network meta-analysis, and moderator-aware meta-regression. We synthesized accuracy data from 120 evaluations covering 10 exam systems in nine languages, identified through comprehensive searches of PubMed, Web of Science, and Institute of Electrical and Electronics Engineers (IEEE) Xplore, covering the period from 2021 to June 2025. The random-effects meta-analysis showed that 13 of 16 models exceeded the 60% passing threshold, with GPT-o1 leading at 95.4%, followed by DeepSeek-R1 (92.0%) and GPT-4o (89.4%). P-score rankings and network meta-analysis confirmed the superior performance of GPT-o1, while GPT-3.5 and LLaMA-13B consistently underperformed. Meta-regression revealed a significant variation in accuracy by model version, exam system, and language, with lower performance in Chinese and Japanese exams and higher performance in German and Peruvian settings. After adjustment for exam and language, GPT-o1 and DeepSeek-R1 achieved similar accuracy, both significantly higher than GPT-4. Model type, exam system, and language explained most of the between-study heterogeneity (R² =88.99%), and sensitivity analyses supported the robustness of the pooled estimates. Overall, several LLMs now approach or exceed the accuracy required to pass standardized medical licensing exams, supporting their potential role in medical education and decision support."