From 95b5857189bcc667f575f332f472d190470f2079 Mon Sep 17 00:00:00 2001 From: Cormac Shannon <> Date: Wed, 28 Oct 2026 18:00:06 +0000 Subject: [PATCH] Add literature review protocol, searches and first snowball round Six seeds snowballed (Semantic Scholar) plus logged keyword searches (OpenAlex, arXiv): 2,127 unique records, 480 title/abstract screened. AI first-pass suggestions are in their own column; decisions are pending. Claude-Session: https://claude.ai/code/session_01S8KN7NuUCm8nXLmy6eDuVv --- 02-literature/README.md | 35 + 02-literature/extraction-table.csv | 59 + 02-literature/protocol.md | 198 ++ .../search/results/A1-arxiv-qknorm.csv | 31 + .../results/A2-arxiv-logit-instability.csv | 10 + .../L1-openalex-llm-for-lit-search.csv | 201 ++ .../L2-openalex-llm-slr-software-eng.csv | 101 + .../search/results/S1-openalex-qknorm.csv | 47 + .../results/S2-openalex-logit-instability.csv | 12 + .../results/S3-openalex-small-scale.csv | 2 + .../results/snowball-back-henry2020.csv | 42 + .../search/results/snowball-back-imu1.csv | 23 + .../results/snowball-back-stability2024.csv | 22 + .../search/results/snowball-back-vit22b.csv | 135 ++ .../results/snowball-back-wortsman2024.csv | 57 + .../search/results/snowball-back-zhai2023.csv | 73 + .../search/results/snowball-fwd-henry2020.csv | 352 +++ .../search/results/snowball-fwd-imu1.csv | 2 + .../results/snowball-fwd-stability2024.csv | 17 + .../search/results/snowball-fwd-vit22b.csv | 973 ++++++++ .../results/snowball-fwd-wortsman2024.csv | 211 ++ .../search/results/snowball-fwd-zhai2023.csv | 222 ++ 02-literature/search/screening.csv | 2128 +++++++++++++++++ 02-literature/search/search-log.csv | 20 + 02-literature/tools/ai_first_pass.py | 139 ++ 02-literature/tools/merge.py | 126 + 02-literature/tools/s2.py | 155 ++ 02-literature/tools/search.py | 154 ++ 28 files changed, 5547 insertions(+) create mode 100644 02-literature/README.md create mode 100644 02-literature/extraction-table.csv create mode 100644 02-literature/protocol.md create mode 100644 02-literature/search/results/A1-arxiv-qknorm.csv create mode 100644 02-literature/search/results/A2-arxiv-logit-instability.csv create mode 100644 02-literature/search/results/L1-openalex-llm-for-lit-search.csv create mode 100644 02-literature/search/results/L2-openalex-llm-slr-software-eng.csv create mode 100644 02-literature/search/results/S1-openalex-qknorm.csv create mode 100644 02-literature/search/results/S2-openalex-logit-instability.csv create mode 100644 02-literature/search/results/S3-openalex-small-scale.csv create mode 100644 02-literature/search/results/snowball-back-henry2020.csv create mode 100644 02-literature/search/results/snowball-back-imu1.csv create mode 100644 02-literature/search/results/snowball-back-stability2024.csv create mode 100644 02-literature/search/results/snowball-back-vit22b.csv create mode 100644 02-literature/search/results/snowball-back-wortsman2024.csv create mode 100644 02-literature/search/results/snowball-back-zhai2023.csv create mode 100644 02-literature/search/results/snowball-fwd-henry2020.csv create mode 100644 02-literature/search/results/snowball-fwd-imu1.csv create mode 100644 02-literature/search/results/snowball-fwd-stability2024.csv create mode 100644 02-literature/search/results/snowball-fwd-vit22b.csv create mode 100644 02-literature/search/results/snowball-fwd-wortsman2024.csv create mode 100644 02-literature/search/results/snowball-fwd-zhai2023.csv create mode 100644 02-literature/search/screening.csv create mode 100644 02-literature/search/search-log.csv create mode 100644 02-literature/tools/ai_first_pass.py create mode 100644 02-literature/tools/merge.py create mode 100644 02-literature/tools/s2.py create mode 100644 02-literature/tools/search.py diff --git a/02-literature/README.md b/02-literature/README.md new file mode 100644 index 0000000..da3f496 --- /dev/null +++ b/02-literature/README.md @@ -0,0 +1,35 @@ +# 02-literature + +| File | What | +|---|---| +| `protocol.md` | **Start here.** Review questions, seeds, search strings, criteria, snowballing, flow counts, LLM-use justification | +| `search/search-log.csv` | Every search and snowball step that was run: date, source, exact query, hit count | +| `search/screening.csv` | All 2,127 de-duplicated records. Fill the `decision` (include/exclude/maybe) and `reason` columns. `ai_proposal` is Claude's first-pass suggestion only | +| `search/results/*.csv` | Raw results, one file per search / snowball step | +| `extraction-table.csv` | One row per proposed include. Bibliographic columns prefilled; the rest is for your full-text reading | +| `references.bib` | BibTeX for the CS6513 paper's references, from arXiv metadata. Import into Zotero | +| `citation-check.md` | Errors found in the CS6513 paper's references | +| `qknorm-research-findings.md` | Earlier (July/Aug) prior-art notes | +| `tools/` | Scripts (Python stdlib only, no installs) | + +## Re-running + +```sh +cd 02-literature +python3 tools/search.py openalex '"qk-norm"|"query-key normalization"' --out S1-openalex-qknorm +python3 tools/search.py arxiv 'abs:"qk norm"' --out A1-arxiv-qknorm +python3 tools/s2.py refs 2309.14322 --out snowball-back-wortsman2024 # backward snowball +python3 tools/s2.py cites 2309.14322 --out snowball-fwd-wortsman2024 # forward snowball +python3 tools/merge.py # rebuilds screening.csv, keeps your decisions +python3 tools/arxiv2bib.py key=2309.14322 >> references.bib +``` + +Semantic Scholar rate-limits unauthenticated use badly. A free key +(semanticscholar.org/product/api) set as `S2_API_KEY` fixes it. + +## Zotero + +1. Install Zotero + the **Better BibTeX** plugin. +2. File → Import → `references.bib`. +3. Better BibTeX → right-click the collection → Export → "Keep updated" → `references.bib`, + so Zotero becomes the source of truth from then on. diff --git a/02-literature/extraction-table.csv b/02-literature/extraction-table.csv new file mode 100644 index 0000000..e7d934a --- /dev/null +++ b/02-literature/extraction-table.csv @@ -0,0 +1,59 @@ +key,first_author,year,title,venue,ai_proposal,close_to_my_rq,model_sizes,data,question_asked,main_claim,method_varied_vs_fixed,qknorm_isolated,mup_used,lr_sweep_points_range,depth_varied,metrics,how_my_study_differs,notes +arxiv:2309.14322,Mitchell Wortsman,2023,Small-scale proxies for large-scale Transformer training instabilities,International Conference on Learning Representations,include [SEED*] LR-sensitivity metric; qk-layernorm at small scale,*,,,,,,,,,,,, +arxiv:2602.02522,George Grigorev,2026,IMU-1: Sample-Efficient Pre-training of Small Language Models,arXiv,"include [SEED*] QK-norm largest win at 70M, bundled with muP",*,,,,,,,,,,,, +doi:10.18653/v1/2020.findings-emnlp.379,Alex Henry,2020,Query-Key Normalization for Transformers,Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP),include [SEED] origin of QK-norm (ACL anthology copy of 2010.04245),,,,,,,,,,,,, +arxiv:2303.06296,Shuangfei Zhai,2023,Stabilizing Transformer Training by Preventing Attention Entropy Collapse,arXiv,include [SEED] attention entropy collapse <-> instability,,,,,,,,,,,,, +arxiv:2302.05442,Mostafa Dehghani,2023,Scaling Vision Transformers to 22 Billion Parameters,International Conference on Machine Learning,include [SEED] QK-norm popularised to stop logit growth at 22B,,,,,,,,,,,,, +arxiv:2410.16682,"Rybakov, Oleg",2024,Methods of improving LLM training stability,arXiv (Cornell University),"include [IC*] QK-norm + softcap -> 1.5x LR before divergence, 830M",*,,,,,,,,,,,, +arxiv:2511.21377,Ben Anson,2025,Controlling changes to attention logits,arXiv,include [IC*] QK-norm vs alternatives for logit control; LR ablations,*,,,,,,,,,,,, +arxiv:2607.18363,Henry Ndubuaku,2026,A Controlled Study of Attention-Only Transformers,arXiv,include [IC*] QK-norm keeps 48-layer stacks trainable: depth evidence (H3),*,,,,,,,,,,,, +arxiv:2610.34582,Harish Santhanalakshmi Ganesan,2026,HobbyLM: A Sparse Mixture-of-Experts Model Family on a Hobby Budget,arXiv,include [IS*] single-variable ablations at 130M incl. QK-norm (HobbyLM),*,,,,,,,,,,,, +arxiv:2610.17969,Bum Jun Kim,2026,Neural Dimensional Analysis: Foundations of Dimensionless and Scale-Consistent Computation,arXiv,include [IA*] compares vs QK-Norm across 16x LR range at 124M,*,,,,,,,,,,,, +doi:10.5281/zenodo.20803082,Shubham Kumar,2026,Gating and Normalization Interact in the Attention Block,Zenodo (CERN European Organization for Nuclear Research),"include [IS*] 29-36M, QK-norm x depth x 3 seeds sweep; Zenodo preprint, check quality",*,,,,,,,,,,,, +arxiv:2606.16310,Yizhou Han,2026,QK-Normed MLA: QK normalization without full key caching,arXiv,include [IC] QK-norm permits higher LR; high-LR stress test vs QK-clip,,,,,,,,,,,,, +arxiv:2603.02170,Jintao Zhang,2026,SageBwd: A Trainable Low-bit Attention,arXiv,include [IC] QK-norm necessary for stable low-bit training at large tokens/step,,,,,,,,,,,,, +arxiv:2602.05006,Ezequiel López-Rubio,2026,Enhanced QKNorm normalization for neural transformers with the Lp norm,arXiv.org,include [IC] Lp-norm generalisation of QKNorm (variant),,,,,,,,,,,,, +arxiv:2503.09543,Oskar van der Wal,2025,PolyPythias: Stability and Outliers across Fifty Language Model Pre-Training Runs,International Conference on Learning Representations,include [IS] stability across 50 small-model seeds: seed-variance precedent,,,,,,,,,,,,, +arxiv:2110.09456,Sam Shleifer,2021,NormFormer: Improved Transformer Pretraining with Extra Normalization,arXiv.org,include [IA] NormFormer: extra normalisation,,,,,,,,,,,,, +arxiv:2312.16903,Sho Takase,2023,Spike No More: Stabilizing the Pre-training of Large Language Models,arXiv.org,include [IA] pretraining loss-spike mitigation,,,,,,,,,,,,, +arxiv:2503.04598,Zhijian Zhuo,2025,HybridNorm: Towards Stable and Efficient Transformer Training via Hybrid Normalization,Neural Information Processing Systems,include [IA] HybridNorm: normalisation placement neighbour,,,,,,,,,,,,, +arxiv:2502.02732,Jeonghoon Kim,2025,Peri-LN: Revisiting Normalization Layer in the Transformer Architecture,International Conference on Machine Learning,include [IA] Peri-LN: normalisation placement neighbour,,,,,,,,,,,,, +arxiv:2507.20534,Kimi Team Yifan Bai,2025,Kimi K2: Open Agentic Intelligence,,include [IA] MuonClip/QK-clip: alternative fix for attention-logit explosion,,,,,,,,,,,,, +arxiv:2602.01212,Marco Chen,2026,SimpleGPT: Improving GPT via A Simple Normalization Strategy,arXiv.org,"include [IA] SimpleGPT normalisation vs LLaMA2+QKNorm, LR ablations",,,,,,,,,,,,, +arxiv:2304.13013,Mitchell Wortsman,2023,Stable and low-precision training for large-scale vision-language models,Neural Information Processing Systems,include [IM] Wortsman et al. follow-up on instability,,,,,,,,,,,,, +arxiv:2410.10781,Xiangming Gu,2024,When Attention Sink Emerges in Language Models: An Empirical View,International Conference on Learning Representations,include [IM] when attention sinks emerge (seed paper ref),,,,,,,,,,,,, +arxiv:2402.17762,Mingjie Sun,2024,Massive Activations in Large Language Models,arXiv.org,include [IM] massive activations -> attention sinks,,,,,,,,,,,,, +arxiv:2402.02098,Han Bao,2024,Self-attention Networks Localize When QK-eigenspectrum Concentrates,International Conference on Machine Learning,include [IM] QK eigenspectrum concentration and entropy collapse,,,,,,,,,,,,, +arxiv:2505.15548,Suvadeep Hajra,2025,Dense Local Dependencies Induce Attention-Logit Explosion and Training Instability During Long-Sequence Transformer Training,arXiv,include [IM] dense local dependencies -> attention-logit explosion,,,,,,,,,,,,, +arxiv:2505.21910,Xianbiao Qi,2025,Taming Transformer Without Using Learning Rate Warmup,arXiv,"include [IM] W_q^T W_k spectral concentration -> entropy collapse, warmup",,,,,,,,,,,,, +arxiv:2505.24333,Alessio Giorlandino,2025,Two failure modes of deep transformers and how to avoid them: a unified theory of signal propagation at initialisation,arXiv,include [IM] rank vs entropy collapse in deep transformers (depth),,,,,,,,,,,,, +doi:10.48448/nht8-e622,Association for Computational Linguistics 2025,2025,Variance Sensitivity Induces Attention Entropy Collapse and Instability in Transformers,Underline Science Inc.,include [IM] variance sensitivity -> attention entropy collapse,,,,,,,,,,,,, +arxiv:2502.05795,Wenfang Sun,2025,The Curse of Depth in Large Language Models,Neural Information Processing Systems,include [IM] depth effects in LLMs (H3 context),,,,,,,,,,,,, +arxiv:2608.14712,Marios Papamichalis,2026,Which Question Is Your Attention Metric Answering? Attention Rows as Compositional Data,,include [IM] how to measure attention entropy properly (metric choice),,,,,,,,,,,,, +arxiv:2002.04745,Ruibin Xiong,2020,On Layer Normalization in the Transformer Architecture,International Conference on Machine Learning,"include [BG] Pre-LN vs Post-LN, warmup and LR (2020)",,,,,,,,,,,,, +arxiv:2203.03466,Greg Yang,2022,Tensor Programs V: Tuning Large Neural Networks via Zero-Shot Hyperparameter Transfer,arXiv.org,include [BG] muP: the confound held OFF,,,,,,,,,,,,, +arxiv:2501.18795,Bowen Yang,2025,Rope to Nope and Back Again: A New Hybrid Attention Strategy,arXiv,"maybe [IC] RoPE/NoPE/QK-norm attention patterns, long context focus",,,,,,,,,,,,, +arxiv:2603.11611,Mohammad Aflah Khan,2026,"Fractional Rotation, Full Potential? Investigating Performance and Convergence of Partial RoPE",arXiv,maybe [IC] NoPE loss spikes mitigated by QK-norm (side finding),,,,,,,,,,,,, +arxiv:2608.30320,Zi-Han Qiu,2026,"On the Design of Qwen3.8-Next Architecture: Evaluation, Efficiency, and Training Stability",,maybe [IC] industry architecture study incl. training stability,,,,,,,,,,,,, +arxiv:2609.14715,Dushyant Rajput,2026,Depth and Scale in the Sub-150M Regime: JugnuLM-53M vs JugnuLM-110M,arXiv,"maybe [IS] sub-150M depth study (QK-norm fixed, not varied)",,,,,,,,,,,,, +doi:10.5281/zenodo.18924690,Hang Lu,2026,Modernizing GPT-2-Scale Models Under a 10B-Token Training Budget,Zenodo (CERN European Organization for Nuclear Research),"maybe [IS] GPT-2-scale modernisation incl. QK-norm; Zenodo, check quality",,,,,,,,,,,,, +arxiv:2507.13338,Laker Newhouse,2025,Training Transformers with Enforced Lipschitz Constants,arXiv,maybe [IA] Lipschitz transformers trained without QK-norm/softcap,,,,,,,,,,,,, +arxiv:2505.22014,Jörg K. H. Franke,2025,Learning in Compact Spaces with Approximately Normalized Transformer,arXiv,maybe [IA] approx. normalised transformer vs GPT+QK-norm convergence,,,,,,,,,,,,, +arxiv:2503.10622,Jia-Cheng Zhu,2025,Transformers without Normalization,Computer Vision and Pattern Recognition,maybe [IA] DyT replaces norm layers,,,,,,,,,,,,, +arxiv:2601.02031,Felix Stollenwerk,2026,Output Embedding Centering for Stable LLM Pretraining,arXiv.org,maybe [IA] output-logit divergence; z-loss / softcap context,,,,,,,,,,,,, +arxiv:2309.17453,Guang-Xuan Xiao,2023,Efficient Streaming Language Models with Attention Sinks,International Conference on Learning Representations,"maybe [IM] attention sinks (inference paper, mechanism background)",,,,,,,,,,,,, +arxiv:2410.13835,Tian-Yu Guo,2024,Active-Dormant Attention Heads: Mechanistically Demystifying Extreme-Token Phenomena in LLMs,arXiv.org,maybe [IM] extreme-token phenomena mechanism,,,,,,,,,,,,, +arxiv:2501.12352,Ke Alexander Wang,2025,Test-time regression: a unifying framework for designing sequence models with associative memory,arXiv,maybe [IM] theory justifying QK-norm (test-time regression view),,,,,,,,,,,,, +arxiv:2502.00919,Stephen Zhang,2025,"Attention Sinks: A 'Catch, Tag, Release' Mechanism for Embeddings",arXiv,maybe [IM] attention sinks persist with QK-norm,,,,,,,,,,,,, +arxiv:2510.22026,Nikita Karagodin,2025,Normalization in Attention Dynamics,Neural Information Processing Systems,maybe [IM] normalisation in attention dynamics,,,,,,,,,,,,, +arxiv:2608.02091,Shuxiao Xie,2026,"One QK Channel, Many Sources: Tracing Low-Precision Attention Collapse",arXiv,maybe [IM] low-precision attention collapse; QK-norm disrupts it,,,,,,,,,,,,, +arxiv:2609.34272,Jun-Lin Chen,2026,Broken Symmetry in BF16 Attention: Why FlashAttention Gradients Blow Up Late in Training,,maybe [IM] BF16 attention gradients; logit growth late in training,,,,,,,,,,,,, +arxiv:2606.28116,Rui-Xuan Huang,2026,Mechanism-Aligned Observability for Training Instability of Large Language Models,,maybe [IM] instability diagnostics; possible metrics,,,,,,,,,,,,, +arxiv:2601.22966,Zihan Qiu,2026,A Unified View of Attention and Residual Sinks: Outlier-Driven Rescaling is Essential for Transformer Training,arXiv (Cornell University),"maybe [IM] attention + residual sinks, outlier rescaling",,,,,,,,,,,,, +arxiv:2602.18849,S. Emadi,2026,Exact Attention Sensitivity and the Geometry of Transformer Stability,arXiv.org,maybe [IM] attention sensitivity and stability geometry,,,,,,,,,,,,, +arxiv:2304.03208,Nolan Dey,2023,Cerebras-GPT: Open Compute-Optimal Language Models Trained on the Cerebras Wafer-Scale Cluster,arXiv.org,maybe [BG] muP in small open models,,,,,,,,,,,,, +arxiv:2407.07972,Rosie Zhao,2024,Deconstructing What Makes a Good Optimizer for Language Models,International Conference on Learning Representations,"maybe [BG] LR stability across optimisers, small LMs",,,,,,,,,,,,, +arxiv:2405.13698,Xi Wang,2024,How to set AdamW's weight decay as you scale model and dataset size,International Conference on Machine Learning,maybe [BG] AdamW hyperparameters vs scale,,,,,,,,,,,,, +arxiv:2510.19093,Atli Kosson,2025,Weight Decay may matter more than muP for Learning Rate Transfer in Practice,arXiv.org,maybe [BG] LR transfer without muP,,,,,,,,,,,,, +arxiv:2505.01618,Nolan Dey,2025,Don't be lazy: CompleteP enables compute-efficient deep transformers,Neural Information Processing Systems,"maybe [BG] depth-wise parametrisation, LR across depth",,,,,,,,,,,,, +arxiv:2608.24814,Zi-Han Liu,2026,Effective Learning Rate Governs Loss Dynamics in Language Model Pretraining,,maybe [BG] effective LR view of loss dynamics,,,,,,,,,,,,, diff --git a/02-literature/protocol.md b/02-literature/protocol.md new file mode 100644 index 0000000..50b07fc --- /dev/null +++ b/02-literature/protocol.md @@ -0,0 +1,198 @@ +# Literature Review Protocol (pared-down SLR) + +*Version 0.1 — drafted 28/10/2026 with AI assistance (see section 9). Status: DRAFT for +Cormac to review and own. `[Your call: …]` marks decisions that are yours to make.* + +Follows a pared-down version of Kitchenham-style SLR guidance, as summarised in +Xiao & Watson (2017, *J. Planning Education and Research*, doi:10.1177/0739456X17723971 — +in `05-course-materials/week2-literature-review/`), with snowballing per +Wohlin (2014, EASE, doi:10.1145/2601248.2601268). + +--- + +## 1. Review questions + +Supervisor advice: ask **how**, not **if** — "how" questions give a scope that can be +answered with a set of findings, rather than a yes/no. + +| ID | Review question (draft) | Feeds | +|---|---|---| +| RQ-L1 | **How** has QK-norm (and related attention-logit control) been evaluated with respect to training stability and learning-rate sensitivity, and at what model scales? | Gap: is <100M covered? | +| RQ-L2 | **How** do reported benefits of QK-norm depend on learning rate, depth and model size? | H1, H2, H3 | +| RQ-L3 | **How** do existing studies isolate QK-norm from confounds (μP, z-loss, soft-capping, optimiser, precision)? | "isolated" claim | +| RQ-L4 | **How** do authors explain *why* QK-norm stabilises training (logit growth, entropy collapse, sinks, kurtosis)? | metrics to log | + +**Thesis RQ (current, from notebook 21/08/2026):** "At <100M parameters, *does* isolated QK-norm +improve training convergence and raise the maximum stable learning rate — and at what learning +rate does it become necessary?" + +`[Your call: possible "how" rewrite — "How does isolated QK-norm change the maximum stable +learning rate and LR-sensitivity of <100M-parameter language models, and how does the learning +rate at which it becomes necessary vary with depth?" Revisit after the extraction table is +filled (supervisor: "review if you want to update/change any of your questions").]` + +## 2. Sources + +| Source | Used for | Notes | +|---|---|---| +| Semantic Scholar Graph API | snowballing (references + citations) | best citation graph for ML preprints; unauthenticated access is heavily rate-limited (a free key helps) | +| OpenAlex API | keyword search | open, free; **no reference lists for arXiv preprints**, so not used for snowballing | +| arXiv API | keyword search | where nearly all of this literature appears first | +| Google Scholar | manual check only | no API; use to spot-check that nothing big is missing `[Your call: run the S1 string by hand and log the top 50]` | + +All automated searches are logged in `search/search-log.csv` (date, source, exact query, +filters, hit count, results file). Raw API responses are cached in `search/.cache/` +(git-ignored) so every search can be re-run and gives the same results. + +## 3. Seed set + +Chosen from the CS6513 lit review (`qknorm-research-findings.md`) as the papers the thesis +depends on most: + +| Key | Paper | Why it is a seed | +|---|---|---| +| henry2020 | Henry et al. 2020, *Query-Key Normalization for Transformers* (Findings of EMNLP) | origin of the technique | +| vit22b | Dehghani et al. 2023, *Scaling Vision Transformers to 22B Parameters* | popularised QK-norm for stability | +| zhai2023 | Zhai et al. 2023, *Preventing Attention Entropy Collapse* (ICML) | mechanism | +| wortsman2024 | Wortsman et al. 2023, *Small-scale proxies for large-scale Transformer training instabilities* | closest method template (LR-sensitivity) | +| imu1 | Grigorev 2026, *IMU-1* | closest small-scale evidence (70M) | +| stability2024 | Rybakov, Chrzanowski, Dykas, Xue et al. 2024, *Methods of improving LLM training stability* (arXiv:2410.16682) | QK-norm → 1.5× LR before divergence | + +Authors of all six seeds checked against arXiv (28/10/2026). + +## 4. Search strings + +Concepts: **(A) technique** × **(B) outcome** × **(C) scale**. Phrases are quoted; `|` = OR. + +| ID | Source | String | +|---|---|---| +| S1 | OpenAlex (title+abstract) | `"qk-norm" \| "qk norm" \| "qk-layernorm" \| "qk layernorm" \| "query-key normalization" \| "query key normalization" \| "qk normalization"` | +| S2 | OpenAlex | (`"attention logit" \| "attention logits" \| "attention entropy collapse" \| "logit growth"`) AND (`"training instability" \| "training stability" \| "loss spike" \| "divergence" \| "learning rate"`) | +| S3 | OpenAlex | (`"small language model(s)" \| "small-scale proxies" \| "small transformer"`) AND (`"learning rate sensitivity" \| "training instability" \| "training stability"`) AND `"normalization"` | +| A1 | arXiv | `abs:"qk norm" OR abs:"qk-norm" OR abs:"query-key normalization" OR abs:"qk normalization" OR abs:"qk-layernorm" OR abs:"qk layernorm"` | +| A2 | arXiv | `(abs:"attention logit" OR abs:"attention logits" OR abs:"entropy collapse") AND (abs:"training instability" OR abs:"learning rate" OR abs:"loss spike")` | +| L1 | OpenAlex, ≥2023 | LLM terms AND (`"systematic literature review" \| "systematic review" \| "literature search" \| "literature screening"`) AND (`"search" \| "screening"`) — *for section 9* | +| L2 | OpenAlex, ≥2023 | LLM terms AND (`"systematic literature review" \| "literature review" \| "snowballing"`) AND `"software engineering"` — *for section 9* | + +S3 returned only 1 hit, so the "small scale" concept is covered mainly by snowballing and by +the extraction table's model-size column rather than by the search string. + +## 5. Inclusion / exclusion criteria + +**Include** if all of: +- **I1** published 2020-01-01 or later (QK-norm was introduced in Oct 2020; nothing earlier is + about it). Pre-2020 *background* (e.g. Vaswani et al. 2017) may be cited, but is not a search result. +- **I2** about transformer models, and addresses QK-norm, attention-logit control, or the + training stability / LR-sensitivity of transformers. +- **I3** English, full text available. + +**Exclude** codes used in `search/screening.csv`: + +| Code | Meaning | +|---|---| +| E1 | applies a transformer to another domain (vision, speech, recommendation, medicine…); QK-norm is just one part of the architecture and its effect isn't studied | +| E2 | inference / systems / hardware only (kernels, KV cache, tensor parallelism) | +| E3 | "entropy collapse" in the RL / policy sense, not attention | +| E4 | duplicate, earlier title of an included paper, or a citation fragment rather than a paper | +| E5 | off-topic on title/abstract | +| E6 | (LLM-search set) uses LLMs as the research *subject*, not as a review tool | + +**Include** codes (why it's in): + +| Code | Meaning | +|---|---| +| SEED | seed paper | +| IC | evaluates QK-norm's effect on stability / LR / loss | +| IA | alternative or complementary stability method, usually compared against QK-norm | +| IM | mechanism: why logits grow, entropy collapse, attention sinks, massive activations | +| IS | small-scale controlled ablation study (method precedent) | +| BG | background / confound (μP, LR transfer, Pre-LN) | +| LLM | evidence on using LLMs to support literature reviews (section 9) | +| `*` | **close to the thesis RQ** — needs "how my study differs" in the extraction table | + +## 6. Snowballing (Wohlin 2014) + +1. **Backward**: every reference of each seed ("what did these guys cite?"). +2. **Forward**: every paper citing each seed. +3. Apply the criteria (I1 automatically; I2 first by keyword filter, then by title/abstract). +4. Each newly *included* paper becomes a seed for the next iteration. Stop when an iteration + adds nothing new. + +**Status:** iteration 1 done for all six seeds (28/10/2026). Iteration 2 not started — +`[Your call]` which of the iteration-1 includes to snowball from (suggest the `*` ones). + +## 7. Screening procedure and flow (iteration 1) + +1. Automatic: de-duplicate by arXiv ID / DOI / title (`tools/merge.py`). +2. Automatic: drop pre-2020 (I1) and records failing a keyword filter for I2. +3. **AI first pass** on title/abstract (`tools/ai_first_pass.py`) → `ai_proposal` column. +4. **Cormac decides** → `decision` column. Disagreements with the AI pass are worth noting — + they're evidence for section 9. +5. Full-text read of includes → extraction table. + +| Stage | QK-norm review | LLM-for-review set | +|---|---|---| +| Records retrieved (all searches + snowballing) | 2,127 unique, after de-duplication | (included in the 2,127) | +| Excluded automatically: pre-2020 | 129 | — | +| Excluded automatically: off-topic keyword filter | 1,518 (both sets) | | +| Title/abstract screened | 237 | 243 | +| AI proposal: include / maybe / exclude | 33 / 25 / 179 | 6 / 6 / 231 | +| **Your decisions** | `[pending]` | `[pending]` | +| Full text read | `[pending]` | `[pending]` | + +## 8. Data extraction + +Template: `extraction-table.csv`. One row per included paper. Columns: +key, citation, year, venue/status, model sizes, data, **question asked**, **main claim**, +method (what's varied / held fixed), QK-norm isolated? (y/n/what bundled), μP used?, +LR sweep (points, range), depth varied?, metrics, close to my RQ (`*`), +**how my study differs (≤2 sentences)**, notes. + +Synthesis questions to answer from the table: +- what questions are people asking across the literature? +- which papers are close to the thesis RQ, and for each: two sentences on why the thesis is different + (e.g. smaller scale, μP off, single variable, laptop cost, dense sweep vs 3 points)? + +## 9. Use of LLMs in this review (and the paper that backs it) + +Supervisor: "if you want to use LLMs for paper search, find a paper that backs up the tactic." + +**What an LLM (Claude Code) was used for:** writing the search scripts; drafting this +protocol and the search strings; a first-pass title/abstract screen (suggestions only, kept in a +separate column); checking citation metadata against arXiv/Crossref. **Not** used for: final +include decisions, the extraction of claims, or writing thesis text. + +**Supporting literature** (all metadata checked against Crossref or arXiv, 28/10/2026): + +- **Kitchenham, Pizard, Madeyski, de Souza Santos, Shepperd (2026)**, *Preliminary Guidelines for + Using and Evaluating GenAI Tools to Support Systematic Literature Reviews*, arXiv:2607.24991. + The GUEST recommendations. Conclude GenAI "requires human oversight and is not currently capable + of unsupervised systematic studies" but offers "cost-effective assistance for some repetitive + tasks". **Main citation for the approach taken here** (human makes every decision; AI suggests). +- Felizardo et al. (2024), *ChatGPT application in SLRs in Software Engineering: an evaluation of its + accuracy to support the selection activity*, ESEM 2024, doi:10.1145/3674805.3686666. +- Felizardo et al. (2025), *On the Difficulties of Conducting and Replicating SLR Studies Using LLMs + in Software Engineering*, WSESE 2025, doi:10.1109/WSESE66602.2025.00010 — why the search log + and fixed scripts matter. +- Wang, Scells, Koopman, Zuccon (2023), *Can ChatGPT Write a Good Boolean Query for Systematic Review + Literature Search?*, SIGIR 2023, doi:10.1145/3539618.3591703 — supports LLM help with search strings. +- Khraisha et al. (2024), *Can large language models replace humans in systematic reviews?*, + Research Synthesis Methods, doi:10.1002/jrsm.1715 — caution: GPT-4 screening performance drops + sharply once chance agreement is accounted for. +- Lieberum et al. (2025), *Large language models for conducting systematic reviews: on the rise, but + not yet ready for use — a scoping review*, J. Clinical Epidemiology, doi:10.1016/j.jclinepi.2025.111746. + +`[Your note: in your own words, why this level of LLM use is defensible given the above.]` + +## 10. Deviations log + +| Date | Planned | Actual | Why | +|---|---|---|---| +| 28/10/2026 | Semantic Scholar for all searching | Keyword searches via OpenAlex + arXiv; S2 for snowballing only | S2 unauthenticated API returned HTTP 429 on search; OpenAlex lacks arXiv reference lists | + +## 11. Defending the review (for the viva / interview) + +"This is how I found these papers": six seeds → one round of forward/backward snowballing on +Semantic Scholar → keyword searches on arXiv and OpenAlex with logged strings → automatic date +and topic filters → title/abstract screening with written reasons → full-text extraction. +Every step is a script plus a log, so it can be re-run. diff --git a/02-literature/search/results/A1-arxiv-qknorm.csv b/02-literature/search/results/A1-arxiv-qknorm.csv new file mode 100644 index 0000000..d3c6441 --- /dev/null +++ b/02-literature/search/results/A1-arxiv-qknorm.csv @@ -0,0 +1,31 @@ +source_id,arxiv,doi,year,title,first_author,n_authors,venue,citations,abstract +2608.09160,2608.09160,,2026,SwiftQK: Fast and Communication-Efficient Tensor Parallelism for Query-Key Normalization,Gyudong Kim,3,arXiv,,"Query-Key Normalization (QK-Norm) improves the training stability and quality of modern Large Language Models (LLMs). However, under Tensor Parallelism (TP), layerwise QK-Norm introduces additional cross-GPU communication because the normalization factor depends on the full hidden vector. We present SwiftQK, a multi-GPU RMSNorm kernel that exchanges only scalar normalization statistics and overlaps the remaining Peer-to-Peer reduction with independent element-wise computation in a deadlock-safe persistent kernel. Evaluations on recent LLMs show that SwiftQK reduces QK-Norm latency by 81.4--93.9% relative to the standard TP QK-Norm using full-vector All-Gather. In end-to-end serving, SwiftQK reduces TPOT on average by 29.5% over the All-Gather-based baseline and by 14.3% over an optimized scalar-aggregation implementation." +2511.21377,2511.21377,,2025,Controlling changes to attention logits,Ben Anson,2,arXiv,,"Stability of neural network weights is critical when training transformer models. The query and key weights are particularly problematic, as they tend to grow large without any intervention. Applying normalization to queries and keys, known as `QK norm', fixes stability issues in practice, but is not always applicable. For example, QK norm is not compatible with Multi Latent Attention (MLA) because QK norm requires full materialization of queries and keys during inference, which is not done in MLA. In this paper we suggest that controlling the changes to logits is important for stability. We show that these changes are controllable by assigning parameter-dependent learning rates to the query and key weights. We find that our cheap intervention allows us to increase the base learning rate of the network, outperform other methods in the MLA setting, and achieve performance competitive with QK norm when using Multi-head Attention." +2606.16310,2606.16310,,2026,QK-Normed MLA: QK normalization without full key caching,Yizhou Han,5,arXiv,,"Query-key (QK) normalization stabilizes attention by controlling the scale of queries and keys before the dot product, but is not immediately compatible with Multi-head Latent Attention (MLA). MLA achieves efficient decoding by caching low-dimensional latent states instead of full keys, whereas post-projection QK RMSNorm appears to require the fully projected key for every cached token. We show this apparent incompatibility is an implementation artifact, not an architectural constraint. RMSNorm decomposes into a static affine weight and a dynamic scalar RMS statistic. The static key-side weight can be absorbed into the MLA query-side projection; the dynamic key statistic reduces to one inverse-RMS scalar per token and KV group. The resulting formulation is exactly equivalent to explicit post-projection QK RMSNorm in exact arithmetic and preserves MLA's latent decode path. In our 400M runs trained for up to 100B tokens, QK-Normed MLA achieves lower training loss and better downstream accuracy than QK clipping, while H800 decode benchmarks show less than 2% latency overhead up to 256k context. These results make QK normalization a practical stabilization option for MLA models without requiring full-key caching." +2404.15702,2404.15702,,2024,Nyonic Technical Report,Junfeng Tian,6,arXiv,,"This report details the development and key achievements of our latest language model designed for custom large language models. The advancements introduced include a novel Online Data Scheduler that supports flexible training data adjustments and curriculum learning. The model's architecture is fortified with state-of-the-art techniques such as Rotary Positional Embeddings, QK-LayerNorm, and a specially crafted multilingual tokenizer to enhance stability and performance. Moreover, our robust training framework incorporates advanced monitoring and rapid recovery features to ensure optimal efficiency. Our Wonton 7B model has demonstrated competitive performance on a range of multilingual and English benchmarks. Future developments will prioritize narrowing the performance gap with more extensively trained models, thereby enhancing the model's real-world efficacy and adaptability.GitHub: \url{https://github.com/nyonicai/nyonic-public}" +2501.18795,2501.18795,,2025,Rope to Nope and Back Again: A New Hybrid Attention Strategy,Bowen Yang,7,arXiv,,"Long-context large language models (LLMs) have achieved remarkable advancements, driven by techniques like Rotary Position Embedding (RoPE) (Su et al., 2023) and its extensions (Chen et al., 2023; Liu et al., 2024c; Peng et al., 2023). By adjusting RoPE parameters and incorporating training data with extended contexts, we can train performant models with considerably longer input sequences. However, existing RoPE-based methods exhibit performance limitations when applied to extended context lengths. This paper presents a comprehensive analysis of various attention mechanisms, including RoPE, No Positional Embedding (NoPE), and Query-Key Normalization (QK-Norm), identifying their strengths and shortcomings in long-context modeling. Our investigation identifies distinctive attention patterns in these methods and highlights their impact on long-context performance, providing valuable insights for architectural design. Building on these findings, we propose a novel architecture featuring a hybrid attention mechanism that integrates global and local attention spans. This design not only surpasses conventional RoPE-based transformer models with full attention in both long and short context tasks but also delivers substantial efficiency gains during training and inference." +2608.02091,2608.02091,,2026,"One QK Channel, Many Sources: Tracing Low-Precision Attention Collapse",Shuxiao Xie,6,arXiv,,"A bfloat16 transformer can train normally, then collapse abruptly. Prior work links collapse to structured attention errors and shows QK normalization disrupts their compounding. Distinct low-precision errors trigger the same collapse, leaving unclear whether each needs a fix at its source or one shared route can be blocked instead. We isolated the fault behind a reproduced GPT-2-class collapse to the streaming-softmax accumulator, where an fp32 streaming core repairs it, and turned it into an assay for moving a controlled error across sources. Using it, we found that errors placed outside attention still drove the same QK spectral runaway, and that correcting only QK kept training stable while the fault stayed active. This is a source-channel dissociation: fault source is not failure channel. It held across tested architectures and scales, and reproduced on a second GPU architecture. As a causal probe, projecting each update off the current QK weights' leading three singular directions held the query projection's largest singular value to 11.1, whereas removing equal energy elsewhere left it at 237: the QK channel causally drives the early runaway. What lets the injected error in is temporal sign-coherence, its per-head sign persisting across steps, not aggregate deviation; once inside, the runaway shows as attention-logit saturation. QK-Guard, a dormant controller, tests this by switching on parameter-free QK normalization at the first monitored threshold crossing. On the runs designated for this test, the QK-local action prevented the failure of each matched or same-configuration unguarded run; on plain GPT-2, all 12 final train and validation losses were within 0.03 nat of same-configuration always-on QK-norm, and both methods ran 60k steps without collapse. Intervention at the QK locus therefore suffices in place of a fix at each source." +2602.02522,2602.02522,,2026,IMU-1: Sample-Efficient Pre-training of Small Language Models,George Grigorev,1,arXiv,,"We present IMU-1, a 430M-parameter language model trained on 72B tokens that approaches the benchmark performance of models trained on 56x more data. We describe a validated training recipe combining recent architectural interventions (QK-norm attention, per-head gating, value residuals, LayerNorm scaling) with optimization advances (NorMuon with cautious weight decay, muP parametrization) and a three-stage training schedule with post-hoc checkpoint EMA. We provide ablations for each component and release code, weights and data to enable reproduction: https://huggingface.co/thepowerfuldeez/imu1_base" +2510.09016,2510.09016,,2025,DiTSinger: Scaling Singing Voice Synthesis with Diffusion Transformer and Implicit Alignment,Zongcai Du,11,arXiv,,"Recent progress in diffusion-based Singing Voice Synthesis (SVS) demonstrates strong expressiveness but remains limited by data scarcity and model scalability. We introduce a two-stage pipeline: a compact seed set of human-sung recordings is constructed by pairing fixed melodies with diverse LLM-generated lyrics, and melody-specific models are trained to synthesize over 500 hours of high-quality Chinese singing data. Building on this corpus, we propose DiTSinger, a Diffusion Transformer with RoPE and qk-norm, systematically scaled in depth, width, and resolution for enhanced fidelity. Furthermore, we design an implicit alignment mechanism that obviates phoneme-level duration labels by constraining phoneme-to-acoustic attention within character-level spans, thereby improving robustness under noisy or uncertain alignments. Extensive experiments validate that our approach enables scalable, alignment-free, and high-fidelity SVS." +2603.02170,2603.02170,,2026,SageBwd: A Trainable Low-bit Attention,Jintao Zhang,8,arXiv,,"Low-bit attention, such as SageAttention, has emerged as an effective approach for accelerating model inference, but its applicability to training remains poorly understood. In prior work, we introduced SageBwd, a trainable INT8 attention that quantizes six of seven attention matrix multiplications while preserving fine-tuning performance. However, SageBwd exhibited a persistent performance gap to full-precision attention (FPA) during pre-training. In this work, we investigate why this gap occurs and demonstrate that SageBwd matches full-precision attention during pretraining. Through experiments and theoretical analysis, we reach a few important insights and conclusions: (i) QK-norm is necessary for stable training at large tokens per step, (ii) quantization errors primarily arise from the backward-pass score gradient dS, (iii) reducing tokens per step enables SageBwd to match FPA performance in pre-training, and (iv) K-smoothing remains essential for training stability, while Q-smoothing provides limited benefit during pre-training." +2603.11611,2603.11611,,2026,"Fractional Rotation, Full Potential? Investigating Performance and Convergence of Partial RoPE",Mohammad Aflah Khan,4,arXiv,,"Rotary Positional Embedding (RoPE) is widely used in transformers to encode relative positional information, yet the effect of applying RoPE to only a subset of hidden dimensions remains underexplored. This design choice can substantially reduce memory usage, especially at long context lengths. We show that partial RoPE achieves up to 10$\times$ memory savings over standard RoPE cache while maintaining comparable final loss. We systematically study its impact on training dynamics and convergence across architectures, model sizes, sequence lengths, and datasets. Our results reveal that: (1) applying RoPE to only a small fraction of dimensions (around 10\%) matches the convergence of full RoPE; (2) these trends remain consistent across settings; and (3) NoPE (No Positional Encoding) can produce loss spikes, which are mitigated by minimal RoPE usage or QK-Norm application. Overall, our findings highlight partial RoPE as an effective trade-off between efficiency and stability." +2602.08612,2602.08612,,2026,OneLive: Dynamically Unified Generative Framework for Live-Streaming Recommendation,Shen Wang,18,arXiv,,"Live-streaming recommender system serves as critical infrastructure that bridges the patterns of real-time interactions between users and authors. Similar to traditional industrial recommender systems, live-streaming recommendation also relies on cascade architectures to support large-scale concurrency. Recent advances in generative recommendation unify the multi-stage recommendation process with Transformer-based architectures, offering improved scalability and higher computational efficiency. However, the inherent complexity of live-streaming prevents the direct transfer of these methods to live-streaming scenario, where continuously evolving content, limited lifecycles, strict real-time constraints, and heterogeneous multi-objectives introduce unique challenges that invalidate static tokenization and conventional model framework. To address these issues, we propose OneLive, a dynamically unified generative recommendation framework tailored for live-streaming scenario. OneLive integrates four key components: (i) A Dynamic Tokenizer that continuously encodes evolving real-time live content fused with behavior signal through residual quantization; (ii) A Time-Aware Gated Attention mechanism that explicitly models temporal dynamics for timely decision making; (iii) An efficient decoder-only generative architecture enhanced with Sequential MTP and QK Norm for stable training and accelerated inference; (iv) A Unified Multi-Objective Alignment Framework reinforces policy optimization for personalized preferences." +2610.17969,2610.17969,,2026,Neural Dimensional Analysis: Foundations of Dimensionless and Scale-Consistent Computation,Bum Jun Kim,4,arXiv,,"Modern Transformers rely on many scale-control mechanisms, including normalization, residual scaling, attention temperatures, loss weighting, and adaptive updates. To explain how these mechanisms maintain scale consistency, we introduce Neural Dimensional Analysis (NDA), a unified framework based on dimensional consistency. We define neural units that specify how tensors rescale and which quantities can be added, just as physical units do in physics. Our rules match residual units, nondimensionalize attention and gate inputs, combine losses across units, and convert gradient units into parameter units. We implement NDA through dimensionless Unit Violation Indices (UVI) for diagnosing scale mismatches within Transformers, a unit-consistent multi-loss objective, and the Gauge-Canonical Transformer (GCT) with normalized coordinates and orthogonalized residual additions. In language-model experiments spanning 51M to 1B parameters, GCT remains stable during training across a broad sixteenfold learning-rate range, outperforming Pre-LN, LLaMA-style, and QK-Norm in 124M validation loss at tuned learning rates. GCT's normalized residual stream also shows improved robustness to quantization. At 1B parameters, calibrated per-tensor int8 quantization increases GCT's validation loss by only 0.009 nats, compared with up to 5.51 nats for the baselines. Vision and physics-informed neural network experiments further confirm NDA's benefits beyond language modeling. These results establish dimensional consistency as a neural network design principle." +2507.13338,2507.13338,,2025,Training Transformers with Enforced Lipschitz Constants,Laker Newhouse,6,arXiv,,"Neural networks are often highly sensitive to input and weight perturbations. This sensitivity has been linked to pathologies such as vulnerability to adversarial examples, divergent training, and overfitting. To combat these problems, past research has looked at building neural networks entirely from Lipschitz components. However, these techniques have not matured to the point where researchers have trained a modern architecture such as a transformer with a Lipschitz certificate enforced beyond initialization. To explore this gap, we begin by developing and benchmarking novel, computationally-efficient tools for maintaining norm-constrained weight matrices. Applying these tools, we are able to train transformer models with Lipschitz bounds enforced throughout training. We find that optimizer dynamics matter: switching from AdamW to Muon improves standard methods -- weight decay and spectral normalization -- allowing models to reach equal performance with a lower Lipschitz bound. Inspired by Muon's update having a fixed spectral norm, we co-design a weight constraint method that improves the Lipschitz vs. performance tradeoff on MLPs and 2M parameter transformers. Our 2-Lipschitz transformer on Shakespeare text reaches validation accuracy 60%. Scaling to 145M parameters, our 10-Lipschitz transformer reaches 21% accuracy on internet text. However, to match the NanoGPT baseline validation accuracy of 39.4%, our Lipschitz upper bound increases to 10^264. Nonetheless, our Lipschitz transformers train without stability measures such as layer norm, QK norm, and logit tanh softcapping." +2609.14715,2609.14715,,2026,Depth and Scale in the Sub-150M Regime: JugnuLM-53M vs JugnuLM-110M,Dushyant Rajput,3,arXiv,,"We scale our conventional sub-150M pretraining recipe from 53.5M to 109.7M parameters, holding the method fixed (Qwen3-style decoder with grouped-query attention, RoPE, SwiGLU, RMSNorm, QK-Norm, and a z-loss; FineWeb-Edu data) and changing only the geometry to a deep-and-thin 23-layer x 576-hidden design. The larger model improves across the board -- BLiMP 78.1 -> 81.3, ARC-Easy 51.4 -> 52.5, WikiText-2 byte-perplexity 2.04 -> 1.95 -- and its 81.3% BLiMP essentially matches GPT-X2-125M (81.28) at about 12% fewer parameters. Notably the 110M model achieves this on fewer training tokens (about 8B vs 12B), so the gain is attributable to capacity and depth, not more data. Both models are deliberately conventional; this report is a clean scaling control and the baseline rung (R0) of an ablation study of what further improves models in this regime. An ablation ladder follows: value residuals (R1) and the Muon optimizer (R2) lift ARC-Easy by a cumulative +3.6 (52.5 -> 56.1) at a near-flat BLiMP and are kept; a diverse data blend (R3) and two logit-distillation settings (R4a/R4b) are not kept -- honest negatives. R3 pins ARC-Easy to FineWeb-Edu's educational filtering rather than raw diversity; distillation from a 1.7B teacher can reach the class-leading ARC-Easy (56.99, matching GPT-X2-125M) but only at a perplexity cost that dialing KD down then erases -- so R2 remains the best kept stack." +2605.18898,2605.18898,,2026,A Two-Parameter Weibull Framework for Diagnosing Transformer Weight Distributions,Tiexin Ding,1,arXiv,,"We apply the Weibull distribution -- a two-parameter family from extreme-value theory -- as a diagnostic framework for element-wise weight magnitude distributions in transformers. At initialization, i.i.d. Gaussian weights give |w| ~ HalfNormal, yielding k ~ 1.20 via middle-80% probability-plot fit (the protocol used throughout this work). This anchor makes k a principled, architecture-independent measuring stick for training dynamics; fitting each weight matrix independently at every layer at every checkpoint enables per-component, per-layer, and per-step diagnostics that aggregate statistics cannot resolve. Applying this framework to 12 model entries spanning 7 architectural families (Pythia, OLMo-1/2, LLaMA-3, Mistral, Qwen2.5/3) reveals three findings. First, FFN modules and the attention output projection W_o -- the Transmission Class -- fall in a narrow k band: median terminal k in [1.186, 1.204] across 12 entries (cross-family CV = 0.51%), shared across SwiGLU/GeLU activations, Pre-LN/QK-Norm placements, and 70M-14B sizes. Second, the attention input projections W_q, W_k -- the Selection Class -- depart from the Weibull family, with severity shaped by storage: separately-stored Q/K (OLMo-1, OLMo-2) yields k in [0.76, 0.99] (deep); GQA models yield k in [1.10, 1.16] (mild); Pythia's merged W_qkv occupies a transitional zone tracking training budget T/tau monotonically. Third, lambda grows substantially during training and scales with sqrt(eta/lambda_wd) within the Pythia family (Pearson r = 0.94, three Transmission kinds), directionally consistent with Fan et al. (2025). The two parameters carry independent information: k labels the functional class, lambda labels training progress. We release npm-weibull-py v0.4 (Python library) and DATABASE_v9_1 at https://github.com/tiexinding/NPM-Weibull-public ." +2605.13989,2605.13989,,2026,VectraYX-Nano: A 42M-Parameter Spanish Cybersecurity Language Model with Curriculum Learning and Native Tool Use,Juan S. Santillana,1,arXiv,,"We present VectraYX-Nano, a 41.95M-parameter decoder-only language model trained from scratch in Spanish for cybersecurity, with a Latin-American regional focus and native tool invocation via the Model Context Protocol (MCP). The model has four contributions. (i) Corpus: VectraYX-Sec-ES, a 170M-token Spanish corpus assembled by an eight-VM distributed pipeline at ~$25 USD of cloud compute and split into three curriculum phases (conversational 42M, cybersecurity 118M, offensive tooling 10M). (ii) Architecture: a 42M Transformer decoder with GQA, QK-Norm, RMSNorm, SwiGLU, RoPE and z-loss, paired with a domain-balanced 16,384-token byte-fallback BPE. (iii) Curriculum with replay across the three phases yields a monotonic loss descent (9.80 -> 3.17 -> 3.00 -> 2.16); after SFT (loss 1.74) the v2 bootstrap-ablation reference attains a conversational gate of 0.775 +/- 0.043 on B5 over N=4 seeds, and a controlled Phase-2 replay sweep over {0,5,10,25,50}% saturates B5 at >=25% replay. (iv) Two empirical findings, both N=4. A controlled bootstrap-corpus ablation across v2 (OpenSubs), v4 (mC4-ES), and v6 (60/25/15 OpenSubs/mC4/Wiki) exposes a loss-versus-register inversion: lower-perplexity bootstraps yield measurably worse conversational behavior (v2 > v4 > v6 on B5 at every paired seed). The B4 (tool-selection) floor of 0.000 is a corpus-density artifact, not a capacity gate: rebalancing the SFT mixture to tool-use ratio 1:21 yields VectraYX-Nano v7, the released headline configuration, reaching B4 = 0.230 +/- 0.052 at 42M while retaining B1 = 0.332 +/- 0.005 and B5 = 0.725 +/- 0.130; a LoRA replication on a 260M from-scratch mid-tier reaches 0.445 +/- 0.201. The released GGUF is 96 MB in F16, runs sub-second TTFT on commodity hardware under llama.cpp, and is, to our knowledge, the first published Spanish-native cybersecurity LLM with end-to-end MCP integration." +2410.07563,2410.07563,,2024,PLaMo-100B: A Ground-Up Language Model Designed for Japanese Proficiency,Preferred Elements,21,arXiv,,"We introduce PLaMo-100B, a large-scale language model designed for Japanese proficiency. The model was trained from scratch using 2 trillion tokens, with architecture such as QK Normalization and Z-Loss to ensure training stability during the training process. Post-training techniques, including Supervised Fine-Tuning and Direct Preference Optimization, were applied to refine the model's performance. Benchmark evaluations suggest that PLaMo-100B performs well, particularly in Japanese-specific tasks, achieving results that are competitive with frontier models like GPT-4. The base model is available at https://huggingface.co/pfnet/plamo-100b." +2502.17092,2502.17092,,2025,Shakti-VLMs: Scalable Vision-Language Models for Enterprise AI,Syed Abdul Gaffar Shakhadri,3,arXiv,,"We introduce Shakti VLM, a family of vision-language models in the capacity of 1B and 4B parameters designed to address data efficiency challenges in multimodal learning. While recent VLMs achieve strong performance through extensive training data, Shakti models leverage architectural innovations to attain competitive results with fewer tokens. Key advancements include QK-Normalization for attention stability, hybrid normalization techniques, and enhanced positional encoding. A three-stage training strategy further optimizes learning efficiency. Evaluations show that Shakti-Shakti-VLM-1B and Shakti-VLM-4B excel in document understanding, Visual Reasoning, OCR extraction, and general multimodal reasoning. Our results highlight that high performance can be achieved through model design and training strategy rather than sheer data volume, making Shakti an efficient solution for enterprise-scale multimodal tasks." +2609.37852,2609.37852,,2026,Delta-Matching: Closing the Final Gap of Native 8-bit Training for LLMs,Haozhan Tang,5,arXiv,,"Reliable FP8 attention remains a barrier to fully native 8-bit large language model training. We derive how forward-backward inconsistencies produce stale delta and empirically show how it distorts training dynamics. Our stale-delta hybrid runs show a modest loss gap at 569M parameters but substantial loss increases and downstream degradation at 1.67B and 5.29B. QK normalization, NoPE (no positional encoding), and lower-learning-rate context extension mitigate or delay degradation without eliminating it. This pattern suggests accumulated optimization error that smaller models and short runs can conceal. We propose Delta-Matching, proving that it restores the softmax gradient's zero-row-sum invariant under the stated numerical assumptions. It enables native block-scaled FP8 in every forward and backward attention-core matmul without architectural changes, smaller global batches, or auxiliary forward outputs. Across tested architectures, scales, and training stages, Delta-Matching matches BF16/FP32 mixed-precision training loss and overall downstream performance. We will release our implementation, trained models, and data recipes." +2606.20027,2606.20027,,2026,QG-MIL: A Gated Transformer Aggregator for Domain-Agnostic Multiple Instance Learning in Medical Imaging,Luca Zedda,7,arXiv,,"Attention-based Multiple Instance Learning aggregators in medical imaging are prone to attention concentration, producing overconfident and unstable predictions. We introduce QG-MIL, a gated transformer aggregator that addresses this through four synergistic architectural components: RMSNorm-based pre-normalization, per-head QK normalization, fine-grained attention output gating, and SwiGLU-style feed-forward modules. Together, these design choices stabilize training and distribute attention more uniformly across instances without auxiliary losses, masking, or multi-stage regularization. We evaluate QG-MIL across six benchmarks spanning whole-slide pathology and cell-level hematology, covering two fundamentally different MIL scales. The best-performing QG-MIL variants outperform leading baselines on all six benchmarks, with an average improvement of +6.1 mean macro F1 points. Attention overlays and attention mass analysis confirm more distributed instance weighting. Ablation studies show that while individual components can match the full model on specific datasets, the QG-MIL design provides the most consistent cross-domain performance and tightest variance when compared to selected baselines. We release a configurable implementation to support reproducibility at: https://github.com/unica-visual-intelligence-lab/QG-MIL" +2505.22014,2505.22014,,2025,Learning in Compact Spaces with Approximately Normalized Transformer,Jörg K. H. Franke,6,arXiv,,"The successful training of deep neural networks requires addressing challenges such as overfitting, numerical instabilities leading to divergence, and increasing variance in the residual stream. A common solution is to apply regularization and normalization techniques that usually require tuning additional hyperparameters. An alternative is to force all parameters and representations to lie on a hypersphere. This removes the need for regularization and increases convergence speed, but comes with additional costs. In this work, we propose a more holistic, approximate normalization via simple scalar multiplications motivated by the tight concentration of the norms of high-dimensional random vectors. Additionally, instead of applying strict normalization for the parameters, we constrain their norms. These modifications remove the need for weight decay and learning rate warm-up as well, but do not increase the total number of normalization layers. Our experiments with transformer architectures show up to 40% faster convergence compared to GPT models with QK normalization, with only 3% additional runtime cost. When deriving scaling laws, we found that our method enables training with larger batch sizes while preserving the favorable scaling characteristics of classic GPT architectures." +2609.23314,2609.23314,,2026,ValueDiff: Value-Geometric KV Cache Eviction for Sink-Suppressed LLMs,Junyoung Park,3,arXiv,,"Modern LLMs with QK-normalization, gated attention, learned attention sinks, or logit softcapping exhibit weaker persistent attention sinks, on which existing KV cache eviction methods primarily rely. We observe that across these models, weaker sinks co-occur with greater value-vector dispersion relative to key-vector dispersion. Motivated by this value-side dispersion, we present ValueDiff, a value-geometric eviction that ranks tokens by the L2 deviation of their value vectors from the cache mean. The same score arises as the minimal-disturbance eviction under a max-entropy assumption about future attention. We evaluate under fixed cache budgets, with eviction at every block boundary during prefill and at every decoding step during generation. On RULER at a tight 2k token budget, ValueDiff retains 88-99% of dense across seven sink-suppressed models (best on 6 out of 7). On LongBench at the 4k budget, ValueDiff averages 92% retention across sink-suppressed models versus 83% for the strongest prior baseline. On MATH-500, ValueDiff is the strongest non-dense method on every sink-suppressed model tested at the 25% cache budget, outperforming prior methods by up to ~20 points on gated-attention models. Across all three benchmarks, value geometry emerges as the more reliable query-invariant eviction signal for sink-suppressed models." +2607.18363,2607.18363,,2026,A Controlled Study of Attention-Only Transformers,Henry Ndubuaku,8,arXiv,,"Feed-forward networks hold two thirds of a transformer's non-embedding parameters, yet the architecture has not received a necessity test that controls parameters, compute, and depth at once. We pretrain attention-only decoder transformers (Simple Attention Networks, SANs) against standard transformers matched separately for parameter count, training FLOPs, and depth (2 to 48 layers), for up to 105B tokens at 6M to 87M parameters. Deleting feed-forward layers in place is costly: the standard transformer leads by 0.47 nats at matched depth and 0.26 nats at matched FLOPs. Reallocating the freed budget into attention depth closes the gap: at matched parameters the difference is 0.006 nats (0.27 percent of loss), reproducible to one part in ten thousand across seed pairs, shrinking across 5B, 30B, and 105B budgets, and holding near 0.02 nats across a 29x size range. Three measurements localize the remaining gap to parametric recall: attention-only models are better on context-grounded answers and worse where knowledge must come from weights. Weight spectra show why: routing matrices (Q/K) crystallize early, content matrices accumulate rank slowly, and removing feed-forward layers relocates this accumulation to the attention output projection. QK-normalization, not feed-forward layers or residual gating, keeps 48-layer attention-only stacks trainable. The deficit concentrates on low-context query prediction and localizes there entirely by the largest budget. A pre-registered test confirms the account: it predicts a 0.02 to 0.05 nat gap on knowledge-dense web text; a matched pair trained on fineweb-edu measures 0.040. Within the tested regime, attention does the rest." +2606.04405,2606.04405,,2026,Low-Rank Decay for Grokking in Scale-Invariant Transformers: A Spectral-Geometric View,Mingyu Li,1,arXiv,,"Modern Transformer architectures frequently employ normalization mechanisms such as RMSNorm and Query-Key Normalization, making parts of the model approximately scale-invariant with respect to weight magnitudes. In this regime, standard Frobenius-norm weight decay acts purely along the radial direction of the weight space and cannot directly simplify the function represented by the normalized layer. We study grokking in small algorithmic tasks through this lens and propose \emph{Low-Rank Decay} (LRD), a nuclear-norm-like spectral regularizer whose subgradient -- the polar factor $UV^\top$ -- retains a tangential component even in the scale-invariant setting. This distinction has a concrete dynamical consequence: after the model memorizes the training set and task gradients vanish, L2 decay can no longer reshape the weight spectrum, whereas LRD continues to compress singular values in an $\ell_1$-like fashion. On modular arithmetic tasks, we find that LRD induces rapid effective-rank collapse in Query/Key matrices and expands the data-fraction boundary at which delayed generalization (grokking) occurs. We further provide a spectral-geometric interpretation through the ``needle-to-fan'' expansion of the nuclear-norm subdifferential near low-rank strata." +2501.12352,2501.12352,,2025,Test-time regression: a unifying framework for designing sequence models with associative memory,Ke Alexander Wang,3,arXiv,,"Sequence models lie at the heart of modern deep learning. However, rapid advancements have produced a diversity of seemingly unrelated architectures, such as Transformers and recurrent alternatives. In this paper, we introduce a unifying framework to understand and derive these sequence models, inspired by the empirical importance of associative recall, the capability to retrieve contextually relevant tokens. We formalize associative recall as a two-step process, memorization and retrieval, casting memorization as a regression problem. Layers that combine these two steps perform associative recall via ``test-time regression'' over its input tokens. Prominent layers, including linear attention, state-space models, fast-weight programmers, online learners, and softmax attention, arise as special cases defined by three design choices: the regression weights, the regressor function class, and the test-time optimization algorithm. Our approach clarifies how linear attention fails to capture inter-token correlations and offers a mathematical justification for the empirical effectiveness of query-key normalization in softmax attention. Further, it illuminates unexplored regions within the design space, which we use to derive novel higher-order generalizations of softmax attention. Beyond unification, our work bridges sequence modeling with classic regression methods, a field with extensive literature, paving the way for developing more powerful and theoretically principled architectures." +2502.00919,2502.00919,,2025,"Attention Sinks: A 'Catch, Tag, Release' Mechanism for Embeddings",Stephen Zhang,3,arXiv,,"Large language models (LLMs) often concentrate their attention on a few specific tokens referred to as attention sinks. Common examples include the first token, a prompt-independent sink, and punctuation tokens, which are prompt-dependent. While the tokens causing the sinks often lack direct semantic meaning, the presence of the sinks is critical for model performance, particularly under model compression and KV-caching. Despite their ubiquity, the function, semantic role, and origin of attention sinks -- especially those beyond the first token -- remain poorly understood. In this work, we conduct a comprehensive investigation demonstrating that attention sinks: catch a sequence of tokens, tag them using a common direction in embedding space, and release them back into the residual stream, where tokens are later retrieved based on the tags they have acquired. Probing experiments reveal these tags carry semantically meaningful information, such as the truth of a statement. These findings extend to reasoning models, where the mechanism spans more heads and explains greater variance in embeddings, or recent models with query-key normalization, where sinks remain just as prevalent. To encourage future theoretical analysis, we introduce a minimal problem which can be solved through the 'catch, tag, release' mechanism, and where it emerges through training." +2610.34582,2610.34582,,2026,HobbyLM: A Sparse Mixture-of-Experts Model Family on a Hobby Budget,Harish Santhanalakshmi Ganesan,3,arXiv,,"We describe HobbyLM, a family of small sparse Mixture-of-Experts (MoE) models trained from scratch with a few hundred to a few thousand dollars of rented GPU time. The core is a decoder-only transformer with fine-grained experts, sigmoid routing with auxiliary-loss-free load balancing, grouped-query attention with per-head QK-normalization, and aMuon/AdamW optimizer split. Each of these choices was selected with a single-variable ablation suite at 130M parameters. On this architecture we train a 500M-parameter model (169M active) on 40B FineWeb tokens and post-train it into a chat model, a masked-diffusion language model, a function-calling and computer-use agent, and a single ""omni"" checkpoint that reads text, images, video, sound and speech through frozen encoders and small projectors. We run these models on a laptop CPU with a from-scratch Rust inference engine. Two of the authors then adopted the architecture without architectural changes to build HobbyLM-1B (1.04B total, 305M active parameters), trained on a 100B-token curated mixture with a high-quality anneal. HobbyLM-1B reaches a 47.5 average on seven zero-shot benchmarks, and its builders also release an instruction-tuned model and an on-device macOS computer-use adapter. Finally, we describe HobbyLM-Image, a 333M-parameter flow-matching diffusion transformer that generates 1024x1024 images in a 32x-compressed latent space, together with a dependency-light CPU implementation of the full text-to-image pipeline. We report what worked, what did not, and where the small-scale ceilings lie. We also state for each component which author designed or built it." +2609.12330,2609.12330,,2026,Unleashing the Power of Equality Saturation for Tensor Program Superoptimization,Qi Zhan,4,arXiv,,"Efficient GPU implementations of tensor programs often require joint optimization of high-level algebraic formulations and low-level execution strategies. However, the resulting search space grows rapidly as transformations combine across operators, making joint optimization difficult to scale. We present EqiForge, a tensor program superoptimizer based on equality saturation. Its unified IR represents high-level tensor expressions and tiled computations in a single expression language. By composing equality rules, EqiForge derives fused implementations such as FlashAttention-style kernels directly from tensor expressions. Early compaction prunes redundant partial programs before completion, while subgraph composition extends the search to larger graphs. Across tensor-program benchmarks, EqiForge achieves a geometric mean speedup of 1.32x and a maximum of 2.74x over the fastest available baseline per configuration. Its attention kernels outperform FlashAttention by up to 1.87x in decode and approach its performance in prefill. EqiForge also discovers new implementations that outperform torch.compile on various Transformer layers, including QK-normalized MLA (3.16x) and mHC (5.84x)." +2610.22330,2610.22330,,2026,Designing Transformers for Continuous Diffusion Language Models,Weijian Luo,5,arXiv,,"Continuous diffusion language models (CDLMs) offer an alternative approach to parallel language generation; however, most existing studies directly use transformer architectures from either causal large language models (LLMs) or Diffusion Transformers (DiTs) for image generation, which were not optimized for CDLMs. In this paper, we conduct a systematic study on Transformer architectures for CDLMs and introduce Language Diffusion Transformer (LangDiT), combining feature and query-key normalization, wider noise conditioning, likelihood-shortcut gradient routing and calibration, and confidence-gated training self-conditioning. LangDiT significantly reduces generation perplexity in matched LangFlow comparisons, establishing a new state of the art in continuous language generation across different model formulations. Using LangFlow as a representative CDLM, LangDiT as a plug-and-play backbone leads to a new state-of-the-art model with a generation perplexity (GenPPL) of 21.72 using 1024 NFEs and 44.72 GenPPL using 128 NFEs on the unconditional generation of 1024-token sequences on the OpenWebText dataset. When adapted to Plaid-style CDLM formulations, LangDiT results in a new leading GenPPL of 11.33 using 1024 denoising NFEs, and 27.51 GenPPL using 128 denoising NFEs. These gains are achieved while retaining each model's underlying representation, training objective, and inference sampler, isolating transformer architecture as a fundamental source of improvement. Our study identifies architectural design as an important source of improvement in continuous diffusion language models." +2604.01683,2604.01683,,2026,Invertible Query-Key Coupling Composes with Attention Mechanisms,Barak Gahtan,2,arXiv,,"Scaled dot-product attention forms its queries and keys as independent linear projections, so the two never interact before the dot product that scores them. We study coupled query-key dynamics, a pre-scoring transformation that evolves each token's query and key jointly through a shared invertible coupling before standard scoring. We realize it as an alternating affine map in the style of real non-volume-preserving flows: the coupling is the identity at initialization, adds a small fraction of parameters per head, and leaves the softmax and surrounding architecture unchanged. We place coupling on top of existing attention methods rather than replacing them, and ask whether that composition helps. On WikiText-103, adding coupling to Differential Attention improves on it at both 150M and 455M parameters. At 455M the gain is significant at sequence length 512 (p=0.003, six seeds), survives a Bonferroni correction and replicates on a held-out test split; it also holds across rotary-embedding training lengths 512, 1024, and 2048. The same additive direction appears when coupling is added to query-key normalization (significant at 150M) and Multi-Token Attention (directional). Matched controls attribute the gain to the joint pre-scoring coupling rather than to added capacity, and show that removing the invertibility guarantee preserves the 455M gain yet is far worse than the base method at 150M, so invertibility is what makes the coupling reliable across scales. On its own, coupling lowers perplexity at 60M and 150M (one-sided Welch tests, p<0.05) but the gain narrows with scale and is not significant at 455M. We relate the construction to the expressivity of coupling flows, use an associative-recall study to map where coupling helps and where it degrades sharp retrieval, and conclude that coupling is most useful in composition with a scoring-stage method rather than as a standalone change." diff --git a/02-literature/search/results/A2-arxiv-logit-instability.csv b/02-literature/search/results/A2-arxiv-logit-instability.csv new file mode 100644 index 0000000..e4a5571 --- /dev/null +++ b/02-literature/search/results/A2-arxiv-logit-instability.csv @@ -0,0 +1,10 @@ +source_id,arxiv,doi,year,title,first_author,n_authors,venue,citations,abstract +2505.15548,2505.15548,,2025,Dense Local Dependencies Induce Attention-Logit Explosion and Training Instability During Long-Sequence Transformer Training,Suvadeep Hajra,1,arXiv,,"Autoregressive transformer language models frequently exhibit training instability when trained on long sequences, particularly under low-precision arithmetic. Although this instability is often accompanied by attention-logit explosion, its underlying cause remains poorly understood. In this work, we present analytical insights and empirical evidence that dense local dependencies are a major contributor to attention-logit explosion. We demonstrate that dense local dependency patterns yield an effectively high-rank attention structure, which the low-rank parameterization of self-attention can only approximate with increasingly large logits as the sequence length grows. This logit inflation ultimately leads to training instability under low-precision arithmetic. We support this explanation through extensive experiments on synthetic and language modeling tasks. Our results consistently show that attention-logit growth increases with sequence length, is mitigated by increasing the attention dimension, and is substantially reduced by explicitly modeling dense local dependencies. Furthermore, we show that this growth is driven by the density of local dependencies rather than by locality alone. More broadly, our findings suggest that explicitly modeling dense local dependencies constitutes an important design principle for developing stable, efficient, and scalable long-context transformer architectures for autoregressive language modeling." +2609.11655,2609.11655,,2026,Musec: MomentUm SpEctral Clipping for Stable Muon-type Training,Zhuanghua Liu,3,arXiv,,"Muon has emerged as a highly effective optimizer for large language model training, often achieving superior convergence and performance compared with the widely adopted Adam and AdamW optimizers. Nevertheless, Muon is prone to training instability due to its spectral flattening, manifested by loss spikes and unbounded growth of model weights. Existing approaches primarily rely on weight or attention-logit clipping, which require architecture-specific modifications and do not directly address instability across all model components. We propose MomentUm SpEctral Clipping (Musec), which replaces Muon's spectral flattening with spectral clipping: rather than setting all singular values of the momentum matrix to approximately one, Musec clips singular values that exceed a threshold while preserving the underlying spectral structure of the momentum. Our strategy provides an optimizer-level, architecture-agnostic mechanism for stabilizing Muon training. Theoretically, we establish convergence guarantees for Musec in nonconvex nonsmooth stochastic optimization. Practically, we develop Soft Musec, an efficient implementation that uses a smooth spectral saturation function approximated by coupled Newton-Schulz iterations. Empirically, Soft Musec consistently improves training stability over existing Muon variants across a wide range of learning rates and model sizes, remaining stable in settings where existing Muon variants diverge while matching their performance under well-tuned configurations." +2303.06296,2303.06296,,2023,Stabilizing Transformer Training by Preventing Attention Entropy Collapse,Shuangfei Zhai,8,arXiv,,"Training stability is of great importance to Transformers. In this work, we investigate the training dynamics of Transformers by examining the evolution of the attention layers. In particular, we track the attention entropy for each attention head during the course of training, which is a proxy for model sharpness. We identify a common pattern across different architectures and tasks, where low attention entropy is accompanied by high training instability, which can take the form of oscillating loss or divergence. We denote the pathologically low attention entropy, corresponding to highly concentrated attention scores, as $\textit{entropy collapse}$. As a remedy, we propose $σ$Reparam, a simple and efficient solution where we reparametrize all linear layers with spectral normalization and an additional learned scalar. We demonstrate that $σ$Reparam successfully prevents entropy collapse in the attention layers, promoting more stable training. Additionally, we prove a tight lower bound of the attention entropy, which decreases exponentially fast with the spectral norm of the attention logits, providing additional motivation for our approach. We conduct experiments with $σ$Reparam on image classification, image self-supervised learning, machine translation, speech recognition, and language modeling tasks. We show that $σ$Reparam provides stability and robustness with respect to the choice of hyperparameters, going so far as enabling training (a) a Vision Transformer {to competitive performance} without warmup, weight decay, layer normalization or adaptive optimizers; (b) deep architectures in machine translation and (c) speech recognition to competitive performance without warmup and adaptive optimizers. Code is available at \url{https://github.com/apple/ml-sigma-reparam}." +2604.00726,2604.00726,,2026,Exploring Silent Data Corruption as a Reliability Challenge in LLM Training,Anton Altenbernd,3,arXiv,,"As Large Language Models (LLMs) scale in size and complexity, the consequences of failures during training become increasingly severe. A major challenge arises from Silent Data Corruption (SDC): hardware-induced faults that bypass system-level detection mechanisms. SDC may behave like benign numerical noise, but can also cause harmful gradient corruption that leads to loss spikes, divergence, or stalled progress. This work provides a controlled study of how intermittent SDC affects LLM pretraining. Using targeted fault injection at the level of GPU matrix-multiply instructions, we characterize the sensitivity of different bit positions, kernel functions, and execution stages. Our analysis shows that locally originating faults can produce impactful corruption, including NaN propagation, short-lived spikes in loss, gradient norm, and attention logits, as well as persistent parameter divergence. Building on the observed corruption signatures, we propose a lightweight detection method that identifies potentially harmful parameter updates. Experiments on LLaMA models with 60M, 350M, and 1.3B parameters demonstrate that recomputing the most recent training step upon detection can effectively mitigate the impact of these events." +2610.11975,2610.11975,,2026,Predicting Cable Dynamics with Physical Attention Bias,Avihai Giuili,4,arXiv,,"Learned simulators for deformable linear objects (DLOs) such as cables have to predict the motion of cables they were not trained on and stay stable over long rollouts. Most of their error occurs where the cable touches itself or the floor. Attention over all pairs of cable segments can represent contact between parts of the cable that are far apart along its length, but attention has no notion of geometry. A cable has two pairwise distances, which agree only while it is straight: the arc-length distance along the cable, which governs elastic forces, and the Euclidean distance in space, which governs contact. We add a physical attention bias, an additive term on the attention logits with a learned rate, and ask which distance it should use. We compare no bias, each distance alone, and both distances on disjoint sets of heads, keeping the rest of the model and the training protocol fixed. A physical bias improves prediction on unseen cables. The gain is largest when attention is the only mechanism that connects distant segments: there, the arc-length bias reduces prediction error by 15% and more than halves the drift in segment length. The Euclidean bias alone stays close to unbiased attention, while assigning both distances across heads is best or near-best on every metric we report. Code and per-run records: https://github.com/avihaig/dlogps." +2609.07148,2609.07148,,2026,Stable-MM-R1: Anchoring Multimodal Reasoning Dynamics via Entropy-Guided Stratification,Yimeng Ye,11,arXiv,,"While Reinforcement Learning (RL) effectively incentivizes reasoning in Large Language Models, current pipelines are hindered by training instability and rapid entropy collapse. These limitations often stem from ""Rollout Silencing"" and low-quality gradient signals in standard sampling procedures. In this work, we propose a robust, data-centric framework to stabilize RL training. We first introduce Potential-Aware Query Mining (PAQM), which filters data dynamically to focus on the ""Distillation Zone""---samples with high potential for capability elicitation. Furthermore, we present Hybrid Stratified Replay (HSR), a novel mechanism that restructures batches by stratifying rollouts based on Path Entropy, a rollout-level confidence proxy, and outcome reward. Within each optimization step, HSR reuses current-policy ""Stability Anchors"" and ""Hard Negatives"" to construct high-contrast optimization groups, then clears its buffers before the next step. This approach mitigates entropy collapse while improving the utilization of learning signals under limited compute. Our method outperforms strong baselines on complex reasoning tasks, offering a principled solution for stable and efficient RL fine-tuning." +2505.21910,2505.21910,,2025,Taming Transformer Without Using Learning Rate Warmup,Xianbiao Qi,8,arXiv,,"Scaling Transformer to a large scale without using some technical tricks such as learning rate warump and using an obviously lower learning rate is an extremely challenging task, and is increasingly gaining more attention. In this paper, we provide a theoretical analysis for the process of training Transformer and reveal the rationale behind the model crash phenomenon in the training process, termed \textit{spectral energy concentration} of ${\bW_q}^{\top} \bW_k$, which is the reason for a malignant entropy collapse, where ${\bW_q}$ and $\bW_k$ are the projection matrices for the query and the key in Transformer, respectively. To remedy this problem, motivated by \textit{Weyl's Inequality}, we present a novel optimization strategy, \ie, making the weight updating in successive steps smooth -- if the ratio $\frac{σ_{1}(\nabla \bW_t)}{σ_{1}(\bW_{t-1})}$ is larger than a threshold, we will automatically bound the learning rate to a weighted multiple of $\frac{σ_{1}(\bW_{t-1})}{σ_{1}(\nabla \bW_t)}$, where $\nabla \bW_t$ is the updating quantity in step $t$. Such an optimization strategy can prevent spectral energy concentration to only a few directions, and thus can avoid malignant entropy collapse which will trigger the model crash. We conduct extensive experiments using ViT, Swin-Transformer and GPT, showing that our optimization strategy can effectively and stably train these Transformers without using learning rate warmup." +2410.23749,2410.23749,,2024,LATST: Are Transformers Necessarily Complex for Time-Series Forecasting,Dizhen Liang,1,arXiv,,"Transformer-based architectures have achieved remarkable success in natural language processing and computer vision. However, their performance in multivariate long-term forecasting often falls short compared to simpler linear baselines. Previous research has identified the traditional attention mechanism as a key factor limiting their effectiveness in this domain. To bridge this gap, we introduce LATST, a novel approach designed to mitigate entropy collapse and training instability common challenges in Transformer-based time series forecasting. We rigorously evaluate LATST across multiple real-world multivariate time series datasets, demonstrating its ability to outperform existing state-of-the-art Transformer models. Notably, LATST manages to achieve competitive performance with fewer parameters than some linear models on certain datasets, highlighting its efficiency and effectiveness." +2505.24333,2505.24333,,2025,Two failure modes of deep transformers and how to avoid them: a unified theory of signal propagation at initialisation,Alessio Giorlandino,2,arXiv,,"Finding the right initialisation for neural networks is crucial to ensure smooth training and good performance. In transformers, the wrong initialisation can lead to one of two failure modes of self-attention layers: rank collapse, where all tokens collapse into similar representations, and entropy collapse, where highly concentrated attention scores lead to training instability. While previous work has studied different scaling regimes for transformers, an asymptotically exact, down-to-the constant prescription for how to initialise transformers has so far been lacking. Here, we provide an analytical theory of signal propagation through deep transformers with self-attention, layer normalisation, skip connections and MLP. Our theory yields a simple algorithm to compute trainability diagrams that identify the correct choice of initialisation hyper-parameters for a given architecture. We overcome the key challenge, an exact treatment of the self-attention layer, by establishing a formal parallel with the Random Energy Model from statistical physics. We also analyse gradients in the backward path and determine the regime where gradients vanish at initialisation. We demonstrate the versatility of our framework through three case studies. Our theoretical framework gives a unified perspective on the two failure modes of self-attention and gives quantitative predictions on the scale of both weights and residual connections that guarantee smooth training." diff --git a/02-literature/search/results/L1-openalex-llm-for-lit-search.csv b/02-literature/search/results/L1-openalex-llm-for-lit-search.csv new file mode 100644 index 0000000..d7853b7 --- /dev/null +++ b/02-literature/search/results/L1-openalex-llm-for-lit-search.csv @@ -0,0 +1,201 @@ +source_id,arxiv,doi,year,title,first_author,n_authors,venue,citations,abstract +W4393119065,,10.1038/s41746-024-01157-x,2024,The ethics of ChatGPT in medicine and healthcare: a systematic review on Large Language Models (LLMs),Joschka Haltaufderheide,2,npj Digital Medicine,393,"With the introduction of ChatGPT, Large Language Models (LLMs) have received enormous attention in healthcare. Despite potential benefits, researchers have underscored various ethical implications. While individual instances have garnered attention, a systematic and comprehensive overview of practical applications currently researched and ethical issues connected to them is lacking. Against this background, this work maps the ethical landscape surrounding the current deployment of LLMs in medicine and healthcare through a systematic review. Electronic databases and preprint servers were queried using a comprehensive search strategy which generated 796 records. Studies were screened and extracted following a modified rapid review approach. Methodological quality was assessed using a hybrid approach. For 53 records, a meta-aggregative synthesis was performed. Four general fields of applications emerged showcasing a dynamic exploration phase. Advantages of using LLMs are attributed to their capacity in data analysis, information provisioning, support in decision-making or mitigating information loss and enhancing information accessibility. However, our study also identifies recurrent ethical concerns connected to fairness, bias, non-maleficence, transparency, and privacy. A distinctive concern is the tendency to produce harmful or convincing but inaccurate content. Calls for ethical guidance and human oversight are recurrent. We suggest that the ethical guidance debate should be reframed to focus on defining what constitutes acceptable human oversight across the spectrum of applications. This involves considering the diversity of settings, varying potentials for harm, and different acceptable thresholds for performance and certainty in healthcare. Additionally, critical inquiry is needed to evaluate the necessity and justification of LLMs' current experimental use." +W4406110856,,10.1073/pnas.2411962122,2025,Transforming literature screening: The emerging role of large language models in systematic reviews,Fernando M. Delgado-Chaves,8,Proceedings of the National Academy of Sciences,86,"Systematic reviews (SR) synthesize evidence-based medical literature, but they involve labor-intensive manual article screening. Large language models (LLMs) can select relevant literature, but their quality and efficacy are still being determined compared to humans. We evaluated the overlap between title- and abstract-based selected articles of 18 different LLMs and human-selected articles for three SR. In the three SRs, 185/4,662, 122/1,741, and 45/66 articles have been selected and considered for full-text screening by two independent reviewers. Due to technical variations and the inability of the LLMs to classify all records, the LLM's considered sample sizes were smaller. However, on average, the 18 LLMs classified 4,294 (min 4,130; max 4,329), 1,539 (min 1,449; max 1,574), and 27 (min 22; max 37) of the titles and abstracts correctly as either included or excluded for the three SRs, respectively. Additional analysis revealed that the definitions of the inclusion criteria and conceptual designs significantly influenced the LLM performances. In conclusion, LLMs can reduce one reviewer´s workload between 33% and 93% during title and abstract screening. However, the exact formulation of the inclusion and exclusion criteria should be refined beforehand for ideal support of the LLMs." +W4403583423,,10.2196/52597,2024,Large Language Models and Empathy: Systematic Review,Vera Sorin,7,Journal of Medical Internet Research,240,"BACKGROUND: Empathy, a fundamental aspect of human interaction, is characterized as the ability to experience another being's emotions within oneself. In health care, empathy is a fundamental for health care professionals and patients' interaction. It is a unique quality to humans that large language models (LLMs) are believed to lack. OBJECTIVE: We aimed to review the literature on the capacity of LLMs in demonstrating empathy. METHODS: We conducted a literature search on MEDLINE, Google Scholar, PsyArXiv, medRxiv, and arXiv between December 2022 and February 2024. We included English-language full-length publications that evaluated empathy in LLMs' outputs. We excluded papers evaluating other topics related to emotional intelligence that were not specifically empathy. The included studies' results, including the LLMs used, performance in empathy tasks, and limitations of the models, along with studies' metadata were summarized. RESULTS: A total of 12 studies published in 2023 met the inclusion criteria. ChatGPT-3.5 (OpenAI) was evaluated in all studies, with 6 studies comparing it with other LLMs such GPT-4, LLaMA (Meta), and fine-tuned chatbots. Seven studies focused on empathy within a medical context. The studies reported LLMs to exhibit elements of empathy, including emotions recognition and emotional support in diverse contexts. Evaluation metric included automatic metrics such as Recall-Oriented Understudy for Gisting Evaluation and Bilingual Evaluation Understudy, and human subjective evaluation. Some studies compared performance on empathy with humans, while others compared between different models. In some cases, LLMs were observed to outperform humans in empathy-related tasks. For example, ChatGPT-3.5 was evaluated for its responses to patients' questions from social media, where ChatGPT's responses were preferred over those of humans in 78.6% of cases. Other studies used subjective readers' assigned scores. One study reported a mean empathy score of 1.84-1.9 (scale 0-2) for their fine-tuned LLM, while a different study evaluating ChatGPT-based chatbots reported a mean human rating of 3.43 out of 4 for empathetic responses. Other evaluations were based on the level of the emotional awareness scale, which was reported to be higher for ChatGPT-3.5 than for humans. Another study evaluated ChatGPT and GPT-4 on soft-skills questions in the United States Medical Licensing Examination, where GPT-4 answered 90% of questions correctly. Limitations were noted, including repetitive use of empathic phrases, difficulty following initial instructions, overly lengthy responses, sensitivity to prompts, and overall subjective evaluation metrics influenced by the evaluator's background. CONCLUSIONS: LLMs exhibit elements of cognitive empathy, recognizing emotions and providing emotionally supportive responses in various contexts. Since social skills are an integral part of intelligence, these advancements bring LLMs closer to human-like interactions and expand their potential use in applications requiring emotional intelligence. However, there remains room for improvement in both the performance of these models and the evaluation strategies used for assessing soft skills." +W4385623654,,10.1101/2023.08.07.23293769,2023,Large Language Models (LLMs) and Empathy – A Systematic Review,Vera Sorin,7,medRxiv,45,"Abstract Purpose Empathy, a cornerstone of human interaction, is a unique quality to humans that Large Language Models (LLMs) are believed to lack. Our study aims to review the literature on the capacity of LLMs in demonstrating empathy Methods We conducted a literature search on MEDLINE up to July 2023. Seven publications ultimately met the inclusion criteria. Results All studies included in this review were published in 2023. All studies but one focused on ChatGPT-3.5 by OpenAI. Only one study evaluated empathy based on objective metrics, and all others used subjective human assessment. The studies reported LLMs to exhibits elements of empathy, including emotions recognition and providing emotionally supportive responses in diverse contexts, most of which were related to healthcare. In some cases, LLMs were observed to outperform humans in empathy-related tasks. Conclusion LLMs demonstrated some aspects of empathy in variable scenarios, mainly related to healthcare. The empathy may be considered “cognitive” empathy. Social skills are a fundamental aspect of intelligence, thus further research is imperative to enhance these skills in AI." +W4410157139,,10.1093/jamia/ocaf063,2025,The emergence of large language models as tools in literature reviews: a large language model-assisted systematic review,Dmitry Scherbakov,5,Journal of the American Medical Informatics Association,154,"OBJECTIVES: This study aims to summarize the usage of large language models (LLMs) in the process of creating a scientific review by looking at the methodological papers that describe the use of LLMs in review automation and the review papers that mention they were made with the support of LLMs. MATERIALS AND METHODS: The search was conducted in June 2024 in PubMed, Scopus, Dimensions, and Google Scholar by human reviewers. Screening and extraction process took place in Covidence with the help of LLM add-on based on the OpenAI GPT-4o model. ChatGPT and Scite.ai were used in cleaning the data, generating the code for figures, and drafting the manuscript. RESULTS: Of the 3788 articles retrieved, 172 studies were deemed eligible for the final review. ChatGPT and GPT-based LLM emerged as the most dominant architecture for review automation (n = 126, 73.2%). A significant number of review automation projects were found, but only a limited number of papers (n = 26, 15.1%) were actual reviews that acknowledged LLM usage. Most citations focused on the automation of a particular stage of review, such as Searching for publications (n = 60, 34.9%) and Data extraction (n = 54, 31.4%). When comparing the pooled performance of GPT-based and BERT-based models, the former was better in data extraction with a mean precision of 83.0% (SD = 10.4) and a recall of 86.0% (SD = 9.8). DISCUSSION AND CONCLUSION: Our LLM-assisted systematic review revealed a significant number of research projects related to review automation using LLMs. Despite limitations, such as lower accuracy of extraction for numeric data, we anticipate that LLMs will soon change the way scientific reviews are conducted." +W4402824743,,10.1093/jamiaopen/ooae098,2024,Literature search sandbox: a large language model that generates search queries for systematic reviews,Gaelen P. Adam,7,JAMIA Open,18,"Objectives: Development of search queries for systematic reviews (SRs) is time-consuming. In this work, we capitalize on recent advances in large language models (LLMs) and a relatively large dataset of natural language descriptions of reviews and corresponding Boolean searches to generate Boolean search queries from SR titles and key questions. Materials and Methods: We curated a training dataset of 10 346 SR search queries registered in PROSPERO. We used this dataset to fine-tune a set of models to generate search queries based on Mistral-Instruct-7b. We evaluated the models quantitatively using an evaluation dataset of 57 SRs and qualitatively through semi-structured interviews with 8 experienced medical librarians. Results: The model-generated search queries had median sensitivity of 85% (interquartile range [IQR] 40%-100%) and number needed to read of 1206 citations (IQR 205-5810). The interviews suggested that the models lack both the necessary sensitivity and precision to be used without scrutiny but could be useful for topic scoping or as initial queries to be refined. Discussion: Future research should focus on improving the dataset with more high-quality search queries, assessing whether fine-tuning the model on other fields, such as the population and intervention, improves performance, and exploring the addition of interactivity to the interface. Conclusions: The datasets developed for this project can be used to train and evaluate LLMs that map review descriptions to Boolean search queries. The models cannot replace thoughtful search query design but may be useful in providing suggestions for key words and the framework for the query." +W4394956892,,10.1111/medu.15402,2024,A systematic review of large language models and their implications in medical education,Harrison Lucas,3,Medical Education,284,"INTRODUCTION: In the past year, the use of large language models (LLMs) has generated significant interest and excitement because of their potential to revolutionise various fields, including medical education for aspiring physicians. Although medical students undergo a demanding educational process to become competent health care professionals, the emergence of LLMs presents a promising solution to challenges like information overload, time constraints and pressure on clinical educators. However, integrating LLMs into medical education raises critical concerns and challenges for educators, professionals and students. This systematic review aims to explore LLM applications in medical education, specifically their impact on medical students' learning experiences. METHODS: A systematic search was performed in PubMed, Web of Science and Embase for articles discussing the applications of LLMs in medical education using selected keywords related to LLMs and medical education, from the time of ChatGPT's debut until February 2024. Only articles available in full text or English were reviewed. The credibility of each study was critically appraised by two independent reviewers. RESULTS: The systematic review identified 166 studies, of which 40 were found by review to be relevant to the study. Among the 40 relevant studies, key themes included LLM capabilities, benefits such as personalised learning and challenges regarding content accuracy. Importantly, 42.5% of these studies specifically evaluated LLMs in a novel way, including ChatGPT, in contexts such as medical exams and clinical/biomedical information, highlighting their potential in replicating human-level performance in medical knowledge. The remaining studies broadly discussed the prospective role of LLMs in medical education, reflecting a keen interest in their future potential despite current constraints. CONCLUSIONS: The responsible implementation of LLMs in medical education offers a promising opportunity to enhance learning experiences. However, ensuring information accuracy, emphasising skill-building and maintaining ethical safeguards are crucial. Continuous critical evaluation and interdisciplinary collaboration are essential for the appropriate integration of LLMs in medical education." +W4401671212,,10.1016/j.csi.2024.103917,2024,Large language models for code completion: A systematic literature review,Rasha Ahmad Husein,3,Computer Standards & Interfaces,72,"Code completion serves as a fundamental aspect of modern software development, improving developers' coding processes. Integrating code completion tools into an Integrated Development Environment (IDE) or code editor enhances the coding process and boosts productivity by reducing errors and speeding up code writing while reducing cognitive load. This is achieved by predicting subsequent tokens, such as keywords, variable names, types, function names, operators, and more. Different techniques can achieve code completion, and recent research has focused on Deep Learning methods, particularly Large Language Models (LLMs) utilizing Transformer algorithms. While several research papers have focused on the use of LLMs for code completion, these studies are fragmented, and there is no systematic overview of the use of LLMs for code completion. Therefore, we aimed to perform a Systematic Literature Review (SLR) study to investigate how LLMs have been applied for code completion so far. We have formulated several research questions to address how LLMs have been integrated for code completion-related tasks and to assess the efficacy of these LLMs in the context of code completion. To achieve this, we retrieved 244 papers from scientific databases using auto-search and specific keywords, finally selecting 23 primary studies based on an SLR methodology for in-depth analysis. This SLR study categorizes the granularity levels of code completion achieved by utilizing LLMs in IDEs, explores the existing issues in current code completion systems, how LLMs address these challenges, and the pre-training and fine-tuning methods employed. Additionally, this study identifies open research problems and outlines future research directions. Our analysis reveals that LLMs significantly enhance code completion performance across several programming languages and contexts, and their capability to predict relevant code snippets based on context and partial input boosts developer productivity substantially." +W4402231520,,10.2139/ssrn.4943759,2024,Accuracy of Large Language Models for Literature Screening in Systematic Reviews and Meta-Analyses,Zhang-Yi Dai,6,SSRN Electronic Journal,6, +W4402193006,,10.2196/57400,2024,Large Language Models for Mental Health Applications: Systematic Review,Zhijun Guo,6,JMIR Mental Health,269,"BACKGROUND: Large language models (LLMs) are advanced artificial neural networks trained on extensive datasets to accurately understand and generate natural language. While they have received much attention and demonstrated potential in digital health, their application in mental health, particularly in clinical settings, has generated considerable debate. OBJECTIVE: This systematic review aims to critically assess the use of LLMs in mental health, specifically focusing on their applicability and efficacy in early screening, digital interventions, and clinical settings. By systematically collating and assessing the evidence from current studies, our work analyzes models, methodologies, data sources, and outcomes, thereby highlighting the potential of LLMs in mental health, the challenges they present, and the prospects for their clinical use. METHODS: Adhering to the PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) guidelines, this review searched 5 open-access databases: MEDLINE (accessed by PubMed), IEEE Xplore, Scopus, JMIR, and ACM Digital Library. Keywords used were (mental health OR mental illness OR mental disorder OR psychiatry) AND (large language models). This study included articles published between January 1, 2017, and April 30, 2024, and excluded articles published in languages other than English. RESULTS: In total, 40 articles were evaluated, including 15 (38%) articles on mental health conditions and suicidal ideation detection through text analysis, 7 (18%) on the use of LLMs as mental health conversational agents, and 18 (45%) on other applications and evaluations of LLMs in mental health. LLMs show good effectiveness in detecting mental health issues and providing accessible, destigmatized eHealth services. However, assessments also indicate that the current risks associated with clinical use might surpass their benefits. These risks include inconsistencies in generated text; the production of hallucinations; and the absence of a comprehensive, benchmarked ethical framework. CONCLUSIONS: This systematic review examines the clinical applications of LLMs in mental health, highlighting their potential and inherent risks. The study identifies several issues: the lack of multilingual datasets annotated by experts, concerns regarding the accuracy and reliability of generated content, challenges in interpretability due to the ""black box"" nature of LLMs, and ongoing ethical dilemmas. These ethical concerns include the absence of a clear, benchmarked ethical framework; data privacy issues; and the potential for overreliance on LLMs by both physicians and patients, which could compromise traditional medical practices. As a result, LLMs should not be considered substitutes for professional mental health services. However, the rapid development of LLMs underscores their potential as valuable clinical aids, emphasizing the need for continued research and development in this area. TRIAL REGISTRATION: PROSPERO CRD42024508617; https://www.crd.york.ac.uk/prospero/display_record.php?RecordID=508617." +W4403087472,,10.2196/22769,2024,Applications and Concerns of ChatGPT and Other Conversational Large Language Models in Health Care: Systematic Review,Leyao Wang,8,Journal of Medical Internet Research,172,"BACKGROUND: The launch of ChatGPT (OpenAI) in November 2022 attracted public attention and academic interest to large language models (LLMs), facilitating the emergence of many other innovative LLMs. These LLMs have been applied in various fields, including health care. Numerous studies have since been conducted regarding how to use state-of-the-art LLMs in health-related scenarios. OBJECTIVE: This review aims to summarize applications of and concerns regarding conversational LLMs in health care and provide an agenda for future research in this field. METHODS: We used PubMed, ACM, and the IEEE digital libraries as primary sources for this review. We followed the guidance of PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) to screen and select peer-reviewed research articles that (1) were related to health care applications and conversational LLMs and (2) were published before September 1, 2023, the date when we started paper collection. We investigated these papers and classified them according to their applications and concerns. RESULTS: Our search initially identified 820 papers according to targeted keywords, out of which 65 (7.9%) papers met our criteria and were included in the review. The most popular conversational LLM was ChatGPT (60/65, 92% of papers), followed by Bard (Google LLC; 1/65, 2% of papers), LLaMA (Meta; 1/65, 2% of papers), and other LLMs (6/65, 9% papers). These papers were classified into four categories of applications: (1) summarization, (2) medical knowledge inquiry, (3) prediction (eg, diagnosis, treatment recommendation, and drug synergy), and (4) administration (eg, documentation and information collection), and four categories of concerns: (1) reliability (eg, training data quality, accuracy, interpretability, and consistency in responses), (2) bias, (3) privacy, and (4) public acceptability. There were 49 (75%) papers using LLMs for either summarization or medical knowledge inquiry, or both, and there are 58 (89%) papers expressing concerns about either reliability or bias, or both. We found that conversational LLMs exhibited promising results in summarization and providing general medical knowledge to patients with a relatively high accuracy. However, conversational LLMs such as ChatGPT are not always able to provide reliable answers to complex health-related tasks (eg, diagnosis) that require specialized domain expertise. While bias or privacy issues are often noted as concerns, no experiments in our reviewed papers thoughtfully examined how conversational LLMs lead to these issues in health care research. CONCLUSIONS: Future studies should focus on improving the reliability of LLM applications in complex health-related tasks, as well as investigating the mechanisms of how LLM applications bring bias and privacy issues. Considering the vast accessibility of LLMs, legal, social, and technical efforts are all needed to address concerns about LLMs to promote, improve, and regularize the application of LLMs in health care." +W4399977125,,10.3389/fpsyt.2024.1422807,2024,Applications of large language models in psychiatry: a systematic review,Mahmud Omar,6,Frontiers in Psychiatry,85,"Background: With their unmatched ability to interpret and engage with human language and context, large language models (LLMs) hint at the potential to bridge AI and human cognitive processes. This review explores the current application of LLMs, such as ChatGPT, in the field of psychiatry. Methods: We followed PRISMA guidelines and searched through PubMed, Embase, Web of Science, and Scopus, up until March 2024. Results: From 771 retrieved articles, we included 16 that directly examine LLMs' use in psychiatry. LLMs, particularly ChatGPT and GPT-4, showed diverse applications in clinical reasoning, social media, and education within psychiatry. They can assist in diagnosing mental health issues, managing depression, evaluating suicide risk, and supporting education in the field. However, our review also points out their limitations, such as difficulties with complex cases and potential underestimation of suicide risks. Conclusion: Early research in psychiatry reveals LLMs' versatile applications, from diagnostic support to educational roles. Given the rapid pace of advancement, future investigations are poised to explore the extent to which these models might redefine traditional roles in mental health care." +W4395688696,,10.1101/2024.04.26.24306390,2024,A Systematic Review of ChatGPT and Other Conversational Large Language Models in Healthcare,Leyao Wang,8,medRxiv,41,"Background: The launch of the Chat Generative Pre-trained Transformer (ChatGPT) in November 2022 has attracted public attention and academic interest to large language models (LLMs), facilitating the emergence of many other innovative LLMs. These LLMs have been applied in various fields, including healthcare. Numerous studies have since been conducted regarding how to employ state-of-the-art LLMs in health-related scenarios to assist patients, doctors, and public health administrators. Objective: This review aims to summarize the applications and concerns of applying conversational LLMs in healthcare and provide an agenda for future research on LLMs in healthcare. Methods: , 2023, the date when we started paper collection and screening. We investigated these papers and classified them according to their applications and concerns. Results: Our search initially identified 820 papers according to targeted keywords, out of which 65 papers met our criteria and were included in the review. The most popular conversational LLM was ChatGPT from OpenAI (60), followed by Bard from Google (1), Large Language Model Meta AI (LLaMA) from Meta (1), and other LLMs (5). These papers were classified into four categories in terms of their applications: 1) summarization, 2) medical knowledge inquiry, 3) prediction, and 4) administration, and four categories of concerns: 1) reliability, 2) bias, 3) privacy, and 4) public acceptability. There are 49 (75%) research papers using LLMs for summarization and/or medical knowledge inquiry, and 58 (89%) research papers expressing concerns about reliability and/or bias. We found that conversational LLMs exhibit promising results in summarization and providing medical knowledge to patients with a relatively high accuracy. However, conversational LLMs like ChatGPT are not able to provide reliable answers to complex health-related tasks that require specialized domain expertise. Additionally, no experiments in our reviewed papers have been conducted to thoughtfully examine how conversational LLMs lead to bias or privacy issues in healthcare research. Conclusions: Future studies should focus on improving the reliability of LLM applications in complex health-related tasks, as well as investigating the mechanisms of how LLM applications brought bias and privacy issues. Considering the vast accessibility of LLMs, legal, social, and technical efforts are all needed to address concerns about LLMs to promote, improve, and regularize the application of LLMs in healthcare." +W4393318738,,10.1186/s12909-024-05239-y,2024,Large language models for generating medical examinations: systematic review,Yaara Artsi,6,BMC Medical Education,126,"BACKGROUND: Writing multiple choice questions (MCQs) for the purpose of medical exams is challenging. It requires extensive medical knowledge, time and effort from medical educators. This systematic review focuses on the application of large language models (LLMs) in generating medical MCQs. METHODS: The authors searched for studies published up to November 2023. Search terms focused on LLMs generated MCQs for medical examinations. Non-English, out of year range and studies not focusing on AI generated multiple-choice questions were excluded. MEDLINE was used as a search database. Risk of bias was evaluated using a tailored QUADAS-2 tool. RESULTS: Overall, eight studies published between April 2023 and October 2023 were included. Six studies used Chat-GPT 3.5, while two employed GPT 4. Five studies showed that LLMs can produce competent questions valid for medical exams. Three studies used LLMs to write medical questions but did not evaluate the validity of the questions. One study conducted a comparative analysis of different models. One other study compared LLM-generated questions with those written by humans. All studies presented faulty questions that were deemed inappropriate for medical exams. Some questions required additional modifications in order to qualify. CONCLUSIONS: LLMs can be used to write MCQs for medical examinations. However, their limitations cannot be ignored. Further study in this field is essential and more conclusive evidence is needed. Until then, LLMs may serve as a supplementary tool for writing medical examinations. 2 studies were at high risk of bias. The study followed the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) guidelines." +W4414761564,,10.1136/rapm-2025-107204,2025,Beware of large language models for scoping literature searches,Ryan David Williams,3,Regional Anesthesia & Pain Medicine,0,"We read with interest the article by Dr Purwel and colleagues on the potential applications of large language models (LLMs) in meta-analyses and systematic reviews.[1][1] While we agree that LLMs could significantly reduce the time required for the laborious components of systematic reviews, we wish" +W4406658975,,10.1038/s43856-024-00717-2,2025,Current applications and challenges in large language models for patient care: a systematic review,Felix Busch,14,Communications Medicine,355,"BACKGROUND: The introduction of large language models (LLMs) into clinical practice promises to improve patient education and empowerment, thereby personalizing medical care and broadening access to medical knowledge. Despite the popularity of LLMs, there is a significant gap in systematized information on their use in patient care. Therefore, this systematic review aims to synthesize current applications and limitations of LLMs in patient care. METHODS: We systematically searched 5 databases for qualitative, quantitative, and mixed methods articles on LLMs in patient care published between 2022 and 2023. From 4349 initial records, 89 studies across 29 medical specialties were included. Quality assessment was performed using the Mixed Methods Appraisal Tool 2018. A data-driven convergent synthesis approach was applied for thematic syntheses of LLM applications and limitations using free line-by-line coding in Dedoose. RESULTS: We show that most studies investigate Generative Pre-trained Transformers (GPT)-3.5 (53.2%, n = 66 of 124 different LLMs examined) and GPT-4 (26.6%, n = 33/124) in answering medical questions, followed by patient information generation, including medical text summarization or translation, and clinical documentation. Our analysis delineates two primary domains of LLM limitations: design and output. Design limitations include 6 second-order and 12 third-order codes, such as lack of medical domain optimization, data transparency, and accessibility issues, while output limitations include 9 second-order and 32 third-order codes, for example, non-reproducibility, non-comprehensiveness, incorrectness, unsafety, and bias. CONCLUSIONS: This review systematically maps LLM applications and limitations in patient care, providing a foundational framework and taxonomy for their implementation and evaluation in healthcare settings." +W4413394768,,10.1109/access.2025.3601206,2025,A Systematic Literature Review of Hallucinations in Large Language Models,Christian Woesle,3,IEEE Access,22,"This review systematically maps research on hallucinations in large language models using a descriptive scheme that links model outputs to four system architectures: unaugmented generation, post-hoc reactive validation, proactive detection-and-mitigation, and fully integrated detection-and-mitigation designs. Our methodology for this systematic review follows the PRISMA guidelines to ensure transparency and reproducibility. We searched IEEE Xplore, ACM Digital Library, and ScienceDirect for studies published between 2015 and January 2025 and extracted 125 peer-reviewed papers across nine application domains. Quantitative analysis shows that question answering and multimodal tasks account for 48% of all papers, whereas software engineering, educational technology, and autonomous systems are underexplored. Although 87.5% of the studies rely on additional reactive or proactive defenses, only 8.8% implement integrated architecture-level safeguards, revealing a critical gap in unified and dynamic architectures. The resulting classification matrix and domain map provide a diagnostic tool for locating blind spots and comparing architectural maturity. Three actionable priorities emerge: develop integrated reasoning-and-verification loops that pre-empt hallucinations; transfer proven causal-intervention and multi-agent validation pipelines to high-stakes, under-represented domains and benchmark them under real conditions; and build modular, cross-domain evaluation frameworks that isolate the contribution of individual mitigation components and support ablation studies. By consolidating fragmented evidence and quantifying architecture-domain imbalances, this review establishes a traceable foundation for engineering reliable, explainable, and domain-adaptable countermeasures to hallucinations in generative language technology." +W4400414272,,10.1001/jamanetworkopen.2024.20496,2024,Performance of a Large Language Model in Screening Citations,Takehiko Oami,3,JAMA Network Open,85,"Importance: Large language models (LLMs) are promising as tools for citation screening in systematic reviews. However, their applicability has not yet been determined. Objective: To evaluate the accuracy and efficiency of an LLM in title and abstract literature screening. Design, Setting, and Participants: This prospective diagnostic study used the data from the title and abstract screening process for 5 clinical questions (CQs) in the development of the Japanese Clinical Practice Guidelines for Management of Sepsis and Septic Shock. The LLM decided to include or exclude citations based on the inclusion and exclusion criteria in terms of patient, population, problem; intervention; comparison; and study design of the selected CQ and was compared with the conventional method for title and abstract screening. This study was conducted from January 7 to 15, 2024. Exposures: LLM (GPT-4 Turbo)-assisted citation screening or the conventional method. Main Outcomes and Measures: The sensitivity and specificity of the LLM-assisted screening process was calculated, and the full-text screening result using the conventional method was set as the reference standard in the primary analysis. Pooled sensitivity and specificity were also estimated, and screening times of the 2 methods were compared. Results: In the conventional citation screening process, 8 of 5634 publications in CQ 1, 4 of 3418 in CQ 2, 4 of 1038 in CQ 3, 17 of 4326 in CQ 4, and 8 of 2253 in CQ 5 were selected. In the primary analysis of 5 CQs, LLM-assisted citation screening demonstrated an integrated sensitivity of 0.75 (95% CI, 0.43 to 0.92) and specificity of 0.99 (95% CI, 0.99 to 0.99). Post hoc modifications to the command prompt improved the integrated sensitivity to 0.91 (95% CI, 0.77 to 0.97) without substantially compromising specificity (0.98 [95% CI, 0.96 to 0.99]). Additionally, LLM-assisted screening was associated with reduced time for processing 100 studies (1.3 minutes vs 17.2 minutes for conventional screening methods; mean difference, -15.25 minutes [95% CI, -17.70 to -12.79 minutes]). Conclusions and Relevance: In this prospective diagnostic study investigating the performance of LLM-assisted citation screening, the model demonstrated acceptable sensitivity and reasonably high specificity with reduced processing time. This novel method could potentially enhance efficiency and reduce workload in systematic reviews." +W4398223047,,10.3390/jcm13113041,2024,Clinical and Surgical Applications of Large Language Models: A Systematic Review,Sophia M. Pressman,6,Journal of Clinical Medicine,106,"Background: Large language models (LLMs) represent a recent advancement in artificial intelligence with medical applications across various healthcare domains. The objective of this review is to highlight how LLMs can be utilized by clinicians and surgeons in their everyday practice. Methods: A systematic review was conducted following the Preferred Reporting Items for Systematic Reviews and Meta-Analyses guidelines. Six databases were searched to identify relevant articles. Eligibility criteria emphasized articles focused primarily on clinical and surgical applications of LLMs. Results: The literature search yielded 333 results, with 34 meeting eligibility criteria. All articles were from 2023. There were 14 original research articles, four letters, one interview, and 15 review articles. These articles covered a wide variety of medical specialties, including various surgical subspecialties. Conclusions: LLMs have the potential to enhance healthcare delivery. In clinical settings, LLMs can assist in diagnosis, treatment guidance, patient triage, physician knowledge augmentation, and administrative tasks. In surgical settings, LLMs can assist surgeons with documentation, surgical planning, and intraoperative guidance. However, addressing their limitations and concerns, particularly those related to accuracy and biases, is crucial. LLMs should be viewed as tools to complement, not replace, the expertise of healthcare professionals." +W4407873728,,10.7326/annals-24-02189,2025,Development of Prompt Templates for Large Language Model–Driven Screening in Systematic Reviews,Christian Cao,19,Annals of Internal Medicine,72,"BACKGROUND: Systematic reviews (SRs) are hindered by the initial rigorous article screen, which delays access to reliable information synthesis. OBJECTIVE: To develop generic prompt templates for large language model (LLM)-driven abstract and full-text screening that can be adapted to different reviews. DESIGN: Diagnostic test accuracy. SETTING: 48 425 citations were tested for abstract screening across 10 SRs. Full-text screening evaluated all 12 690 freely available articles from the original search. Prompt development used the GPT4-0125-preview model (OpenAI). PARTICIPANTS: None. MEASUREMENTS: Large language models were prompted to include or exclude articles based on SR eligibility criteria. Model outputs were compared with original SR author decisions after full-text screening to evaluate performance (accuracy, sensitivity, and specificity). RESULTS: Optimized prompts using GPT4-0125-preview achieved a weighted sensitivity of 97.7% (range, 86.7% to 100%) and specificity of 85.2% (range, 68.3% to 95.9%) in abstract screening and weighted sensitivity of 96.5% (range, 89.7% to 100.0%) and specificity of 91.2% (range, 80.7% to 100%) in full-text screening across 10 SRs. In contrast, zero-shot prompts had poor sensitivity (49.0% abstract, 49.1% full-text). Across LLMs, Claude-3.5 (Anthropic) and GPT4 variants had similar performance, whereas Gemini Pro (Google) and GPT3.5 (OpenAI) models underperformed. Direct screening costs for 10 000 citations differed substantially: Where single human abstract screening was estimated to require more than 83 hours and $1666.67 USD, our LLM-based approach completed screening in under 1 day for $157.02 USD. LIMITATIONS: Further prompt optimizations may exist. Retrospective study. Convenience sample of SRs. Full-text screening evaluations were limited to free PubMed Central full-text articles. CONCLUSION: A generic prompt for abstract and full-text screening achieving high sensitivity and specificity that can be adapted to other SRs and LLMs was developed. Our prompting innovations may have value to SR investigators and researchers conducting similar criteria-based tasks across the medical sciences. PRIMARY FUNDING SOURCE: None." +W7128199163,,10.5220/0013562900003967,2025,Evaluating Large Language Models for Literature Screening: A Systematic Review of Sensitivity and Workload Reduction,Elias Sandner,8,"International Conference on Data Science, Technology and Applications (DATA)",4, +W4407764671,,10.2196/67488,2025,Accuracy of Large Language Models for Literature Screening in Thoracic Surgery: Diagnostic Study,Zhang-Yi Dai,7,Journal of Medical Internet Research,29,"BACKGROUND: Systematic reviews and meta-analyses rely on labor-intensive literature screening. While machine learning offers potential automation, its accuracy remains suboptimal. This raises the question of whether emerging large language models (LLMs) can provide a more accurate and efficient approach. OBJECTIVE: This paper evaluates the sensitivity, specificity, and summary receiver operating characteristic (SROC) curve of LLM-assisted literature screening. METHODS: We conducted a diagnostic study comparing the accuracy of LLM-assisted screening versus manual literature screening across 6 thoracic surgery meta-analyses. Manual screening by 2 investigators served as the reference standard. LLM-assisted screening was performed using ChatGPT-4o (OpenAI) and Claude-3.5 (Anthropic) sonnet, with discrepancies resolved by Gemini-1.5 pro (Google). In addition, 2 open-source, machine learning-based screening tools, ASReview (Utrecht University) and Abstrackr (Center for Evidence Synthesis in Health, Brown University School of Public Health), were also evaluated. We calculated sensitivity, specificity, and 95% CIs for the title and abstract, as well as full-text screening, generating pooled estimates and SROC curves. LLM prompts were revised based on a post hoc error analysis. RESULTS: LLM-assisted full-text screening demonstrated high pooled sensitivity (0.87, 95% CI 0.77-0.99) and specificity (0.96, 95% CI 0.91-0.98), with the area under the curve (AUC) of 0.96 (95% CI 0.94-0.97). Title and abstract screening achieved a pooled sensitivity of 0.73 (95% CI 0.57-0.85) and specificity of 0.99 (95% CI 0.97-0.99), with an AUC of 0.97 (95% CI 0.96-0.99). Post hoc revisions improved sensitivity to 0.98 (95% CI 0.74-1.00) while maintaining high specificity (0.98, 95% CI 0.94-0.99). In comparison, the pooled sensitivity and specificity of ASReview tool-assisted screening were 0.58 (95% CI 0.53-0.64) and 0.97 (95% CI 0.91-0.99), respectively, with an AUC of 0.66 (95% CI 0.62-0.70). The pooled sensitivity and specificity of Abstrackr tool-assisted screening were 0.48 (95% CI 0.35-0.62) and 0.96 (95% CI 0.88-0.99), respectively, with an AUC of 0.78 (95% CI 0.74-0.82). A post hoc meta-analysis revealed comparable effect sizes between LLM-assisted and conventional screening. CONCLUSIONS: LLMs hold significant potential for streamlining literature screening in systematic reviews, reducing workload without sacrificing quality. Importantly, LLMs outperformed traditional machine learning-based tools (ASReview and Abstrackr) in both sensitivity and AUC values, suggesting that LLMs offer a more accurate and efficient approach to literature screening." +W7211887844,,10.17802/2306-1278-2026-15-4-153-163,2026,USING LARGE LANGUAGE MODELS FOR LITERATURE SEARCH IN CARDIOVASCULAR SURGERY SYSTEMATIC REVIEWS AND META-ANALYSES,A. S. Shatskiy,4,Complex Issues of Cardiovascular Diseases,0,"Highlights Large Language Models (LLMs) enable automation of initial abstract screening in systematic reviews, significantly reducing manual workload. The effectiveness of LLMs heavily depends on prompt engineering, which must clearly translate inclusion and exclusion criteria into actionable instructions. Two prompt design strategies are presented and illustrated: a “lenient” approach (maximizing recall) and a “strict” approach (reducing false positives). A practical workflow for downloading and processing PubMed abstracts using LLMs (e.g., DeepSeek, GPT‑4o, Perplexity) is demonstrated. Key limitations of LLMs – hallucinations, sensitivity to phrasing, and training‑data biases — are discussed, emphasizing the need for expert validation. A two‑stage screening strategy is recommended: first a lenient pass, followed by a stricter filter, with subsequent manual verification of relevant articles. Prompt engineering is framed as an iterative, domain‑specific art rather than a one‑size‑fits‑all procedure, requiring continuous refinement for each research question. Abstract Large Language Models have evolved into powerful tools for automating the primary screening of articles’ abstracts in systematic reviews, enabling a significant reduction in manual labor. The article presents a comprehensive review of prompt engineering principles, demonstrating how traditional meta-analysis criteria can be transformed into clear instructions for artificial intelligence. Using case studies in coronary artery bypass grafting and congenital heart disease surgery, we illustrate the impact of prompt formulation on the comprehensiveness, accuracy, and overall efficiency of literature screening. Furthermore, typical errors are discussed, and the ongoing necessity of expert oversight to minimize hallucinations and biases inherent in artificial intelligence conclusions is emphasized. Ultimately, systematic prompt engineering combined with expert evaluation allows researchers in cardiovascular surgery to optimize the search for meta-analysis sources based on Large Language Models, ensuring a faster and more comprehensive synthesis of evidence and methods for processing primary material." +W4410169548,,10.3390/biomedinformatics5020025,2025,Performance Comparison of Large Language Models for Efficient Literature Screening,Maria Teresa Colangelo,5,BioMedInformatics,6,"Background: Systematic reviewers face a growing body of biomedical literature, making early-stage article screening increasingly time-consuming. In this study, we assessed six large language models (LLMs)—OpenHermes, Flan T5, GPT-2, Claude 3 Haiku, GPT-3.5 Turbo, and GPT-4o—for their ability to identify randomized controlled trials (RCTs) in datasets of increasing difficulty. Methods: We first retrieved articles from PubMed and used all-mpnet-base-v2 to measure semantic similarity to known target RCTs, stratifying the collection into quartiles of descending relevance. Each LLM then received either verbose or concise prompts to classify articles as “Accepted” or “Rejected”. Results: Claude 3 Haiku, GPT-3.5 Turbo, and GPT-4o consistently achieved high recall, though their precision varied in the quartile with the highest similarity, where false positives increased. By contrast, smaller or older models struggled to balance sensitivity and specificity, with some over-including irrelevant studies or missing key articles. Importantly, multi-stage prompts did not guarantee performance gains for weaker models, whereas single-prompt approaches proved effective for advanced LLMs. Conclusions: These findings underscore that both model capability and prompt design strongly affect classification outcomes, suggesting that newer LLMs, if properly guided, can substantially expedite systematic reviews." +W4405623173,,10.2196/66648,2024,Large Language Models in Gastroenterology: Systematic Review,Eun Jeong Gong,8,Journal of Medical Internet Research,37,"BACKGROUND: As health care continues to evolve with technological advancements, the integration of artificial intelligence into clinical practices has shown promising potential to enhance patient care and operational efficiency. Among the forefront of these innovations are large language models (LLMs), a subset of artificial intelligence designed to understand, generate, and interact with human language at an unprecedented scale. OBJECTIVE: This systematic review describes the role of LLMs in improving diagnostic accuracy, automating documentation, and advancing specialist education and patient engagement within the field of gastroenterology and gastrointestinal endoscopy. METHODS: Core databases including MEDLINE through PubMed, Embase, and Cochrane Central registry were searched using keywords related to LLMs (from inception to April 2024). Studies were included if they satisfied the following criteria: (1) any type of studies that investigated the potential role of LLMs in the field of gastrointestinal endoscopy or gastroenterology, (2) studies published in English, and (3) studies in full-text format. The exclusion criteria were as follows: (1) studies that did not report the potential role of LLMs in the field of gastrointestinal endoscopy or gastroenterology, (2) case reports and review papers, (3) ineligible research objects (eg, animals or basic research), and (4) insufficient data regarding the potential role of LLMs. Risk of Bias in Non-Randomized Studies-of Interventions was used to evaluate the quality of the identified studies. RESULTS: Overall, 21 studies on the potential role of LLMs in gastrointestinal disorders were included in the systematic review, and narrative synthesis was done because of heterogeneity in the specified aims and methodology in each included study. The overall risk of bias was low in 5 studies and moderate in 16 studies. The ability of LLMs to spread general medical information, offer advice for consultations, generate procedure reports automatically, or draw conclusions about the presumptive diagnosis of complex medical illnesses was demonstrated by the systematic review. Despite promising benefits, such as increased efficiency and improved patient outcomes, challenges related to data privacy, accuracy, and interdisciplinary collaboration remain. CONCLUSIONS: We highlight the importance of navigating these challenges to fully leverage LLMs in transforming gastrointestinal endoscopy practices. TRIAL REGISTRATION: PROSPERO 581772; https://www.crd.york.ac.uk/prospero/." +W7138936899,,10.47989/ir31iconf64265,2026,Collaborating with large language models in literature screening for a systematic review of college students’ GenAI literacy,Wonchan Choi,5,Information Research,0,"Introduction. Literature screening is among the most time- and labour-intensive phases of systematic literature reviews (SLRs). Although large language models (LLMs) have been explored for screening, prior work has focused on medical and environmental domains and mainly benchmarked LLMs against human coders, offering limited guidance on collaborative integration in SLRs. Method. This study evaluated 12 GPT model–prompt configurations in an SLR of 1,616 publications. Two human coders screened a 10% sample (n = 162) to create a gold standard for model comparison. Performance was assessed using balanced accuracy, recall for inclusion, time efficiency, and cost efficiency. Disagreements between humans and models were analysed. Analysis. Bootstrap tests compared performance across configurations. Open coding identified error types in human–model disagreements. Group discussion resolved discrepancies. Results. GPT-5 Zeroshot and GPT-4o-mini Fewshot achieved the highest performance (accuracy = 0.990 and 0.946; recall = 1.000 and 0.933). GPT-4o-mini was faster and cheaper but more prone to overly rigid rule application. Error analysis identified 10 mismatches, leading to two corrections of human miscoding. Conclusion. LLM-assisted screening can reduce workload, improve efficiency, and correct human errors in SLR. Practical guidelines for prompt design and confidence thresholds can position LLMs as collaborative tools in SLRs." +W4404402066,,10.2196/59607,2024,Examining the Role of Large Language Models in Orthopedics: Systematic Review,Cheng Zhang,8,Journal of Medical Internet Research,64,"BACKGROUND: Large language models (LLMs) can understand natural language and generate corresponding text, images, and even videos based on prompts, which holds great potential in medical scenarios. Orthopedics is a significant branch of medicine, and orthopedic diseases contribute to a significant socioeconomic burden, which could be alleviated by the application of LLMs. Several pioneers in orthopedics have conducted research on LLMs across various subspecialties to explore their performance in addressing different issues. However, there are currently few reviews and summaries of these studies, and a systematic summary of existing research is absent. OBJECTIVE: The objective of this review was to comprehensively summarize research findings on the application of LLMs in the field of orthopedics and explore the potential opportunities and challenges. METHODS: PubMed, Embase, and Cochrane Library databases were searched from January 1, 2014, to February 22, 2024, with the language limited to English. The terms, which included variants of ""large language model,"" ""generative artificial intelligence,"" ""ChatGPT,"" and ""orthopaedics,"" were divided into 2 categories: large language model and orthopedics. After completing the search, the study selection process was conducted according to the inclusion and exclusion criteria. The quality of the included studies was assessed using the revised Cochrane risk-of-bias tool for randomized trials and CONSORT-AI (Consolidated Standards of Reporting Trials-Artificial Intelligence) guidance. Data extraction and synthesis were conducted after the quality assessment. RESULTS: A total of 68 studies were selected. The application of LLMs in orthopedics involved the fields of clinical practice, education, research, and management. Of these 68 studies, 47 (69%) focused on clinical practice, 12 (18%) addressed orthopedic education, 8 (12%) were related to scientific research, and 1 (1%) pertained to the field of management. Of the 68 studies, only 8 (12%) recruited patients, and only 1 (1%) was a high-quality randomized controlled trial. ChatGPT was the most commonly mentioned LLM tool. There was considerable heterogeneity in the definition, measurement, and evaluation of the LLMs' performance across the different studies. For diagnostic tasks alone, the accuracy ranged from 55% to 93%. When performing disease classification tasks, ChatGPT with GPT-4's accuracy ranged from 2% to 100%. With regard to answering questions in orthopedic examinations, the scores ranged from 45% to 73.6% due to differences in models and test selections. CONCLUSIONS: LLMs cannot replace orthopedic professionals in the short term. However, using LLMs as copilots could be a potential approach to effectively enhance work efficiency at present. More high-quality clinical trials are needed in the future, aiming to identify optimal applications of LLMs and advance orthopedics toward higher efficiency and precision." +W4394890393,,10.1101/2024.04.15.24305869,2024,A Systematic Review of Testing and Evaluation of Healthcare Applications of Large Language Models (LLMs),Suhana Bedi,19,medRxiv,48,"1 Abstract Importance Large Language Models (LLMs) can assist in a wide range of healthcare-related activities. Current approaches to evaluating LLMs make it difficult to identify the most impactful LLM application areas. Objective To summarize the current evaluation of LLMs in healthcare in terms of 5 components: evaluation data type, healthcare task, Natural Language Processing (NLP)/Natural Language Understanding (NLU) task, dimension of evaluation, and medical specialty. Data Sources A systematic search of PubMed and Web of Science was performed for studies published between 01-01-2022 and 02-19-2024. Study Selection Studies evaluating one or more LLMs in healthcare. Data Extraction and Synthesis Three independent reviewers categorized 519 studies in terms of data used in the evaluation, the healthcare tasks (the what) and the NLP/NLU tasks (the how) examined, the dimension(s) of evaluation, and the medical specialty studied. Results Only 5% of reviewed studies utilized real patient care data for LLM evaluation. The most popular healthcare tasks were assessing medical knowledge (e.g. answering medical licensing exam questions, 44.5%), followed by making diagnoses (19.5%), and educating patients (17.7%). Administrative tasks such as assigning provider billing codes (0.2%), writing prescriptions (0.2%), generating clinical referrals (0.6%) and clinical notetaking (0.8%) were less studied. For NLP/NLU tasks, the vast majority of studies examined question answering (84.2%). Other tasks such as summarization (8.9%), conversational dialogue (3.3%), and translation (3.1%) were infrequent. Almost all studies (95.4%) used accuracy as the primary dimension of evaluation; fairness, bias and toxicity (15.8%), robustness (14.8%), deployment considerations (4.6%), and calibration and uncertainty (1.2%) were infrequently measured. Finally, in terms of medical specialty area, most studies were in internal medicine (42%), surgery (11.4%) and ophthalmology (6.9%), with nuclear medicine (0.6%), physical medicine (0.4%) and medical genetics (0.2%) being the least represented. Conclusions and Relevance Existing evaluations of LLMs mostly focused on accuracy of question answering for medical exams, without consideration of real patient care data. Dimensions like fairness, bias and toxicity, robustness, and deployment considerations received limited attention. To draw meaningful conclusions and improve LLM adoption, future studies need to establish a standardized set of LLM applications and evaluation dimensions, perform evaluations using data from routine care, and broaden testing to include administrative tasks as well as multiple medical specialties. Key Points Question How are healthcare applications of large language models (LLMs) currently evaluated? Findings Studies rarely used real patient care data for LLM evaluation. Administrative tasks such as generating provider billing codes and writing prescriptions were understudied. Natural Language Processing (NLP)/Natural Language Understanding (NLU) tasks like summarization, conversational dialogue, and translation were infrequently explored. Accuracy was the predominant dimension of evaluation, while fairness, bias and toxicity assessments were neglected. Evaluations in specialized fields, such as nuclear medicine and medical genetics were rare. Meaning Current LLM assessments in healthcare remain shallow and fragmented. To draw concrete insights on their performance, evaluations need to use real patient care data across a broad range of healthcare and NLP/NLU tasks and medical specialties with standardized dimensions of evaluation." +W4411259709,,10.1101/2025.06.13.25329541,2025,Automation of Systematic Reviews with Large Language Models,Christian Cao,36,medRxiv,33,"Abstract Importance Systematic reviews (SRs) inform evidence-based decision making. Yet, many take over a year to complete, are labor intensive, prone to human error, and face reproducibility challenges; thus limiting access to timely and reliable information. Objective To validate a large language model (LLM)-based workflow (otto-SR) to automate three of the most labour intensive tasks in performing SR’s: article screening, data extraction, and risk of bias assessment; and to assess its feasibility in rapidly updating existing reviews. Design, setting, and participants We conducted a validation study in four phases, with direct benchmarking against graduate-level human researchers in phases 1 and 2. Phase 1: article screening performance was measured across 32,357 citations from 5 systematic reviews. The reference standard consisted of the original reviews’ screening decisions after full-text screening. Phase 2: data extraction performance was measured across 4,495 data points from 495 studies in 7 reviews. Phase 3: risk of bias assessment (ROB2, Newcastle-Ottawa, QUADAS2) performance was measured across 345 studies from 12 reviews. Reference standards for Phase 2 and Phase 3 were created after blinded adjudication of the original review extraction and RoB assessments. Phase 4: otto-SR was used to reproduce and update the primary analysis from an issue of Cochrane reviews (n=12 reviews, 146,276 citations), with analytical comparisons to the original meta-analyzed findings. All discrepancies underwent dual human review. Results otto-SR showed high performance in phase 1 article screening ( otto-SR : 96.7% sensitivity, 97.9% specificity; human: 81.7% sensitivity, 98.1% specificity) and phase 2 data extraction ( otto-SR : 93.1% accuracy; human: 79.7% accuracy). In phase 3, otto-SR demonstrated high interrater reliability for risk of bias judgements (ROB2 0.98, Newcastle-Ottawa 0.95, QUADAS2 0.74; Gwet AC2). In phase 4, otto-SR , reproduced and updated the primary analysis from an issue of Cochrane reviews. Across Cochrane reviews, otto-SR incorrectly excluded a median of 0 studies (IQR 0 to 0.25), and found nearly twice as many eligible studies compared to the original authors (n= 114 vs. 64). Meta-analyses based on otto-SR generated screening and extraction outputs, subsequently verified through dual human review, yielded newly statistically significant effect estimates in 2 reviews and negated significance in 1 review. Conclusions and relevance LLMs have high performance in article screening, data extraction, and risk of bias assessments. They can rapidly reproduce and update existing systematic reviews, laying the foundation for automated, scalable, and reliable evidence synthesis." +W7133133003,,10.17605/osf.io/56d8j,2025,Performance of Large Language Models in Automated Medical Literature Screening: A Systematic Review and Meta-analysis,Chenggong Xie,1,,0,"This project aims to systematically evaluate the performance, reliability, and heterogeneity of large language models (LLMs) in the automated screening of medical literature. As the volume of scientific publications continues to grow exponentially, the screening phase of systematic reviews has become increasingly time-consuming and labor-intensive. Recent advances in artificial intelligence—particularly large language models such as GPT-series, LLaMA, Claude, Gemini, Copilot, DeepSeek, and Qwen—have shown the potential to automate key steps of evidence synthesis. However, the accuracy, consistency, and generalizability of these models across different screening tasks and clinical domains remain unclear. This study will conduct a comprehensive systematic review and meta-analysis to summarize existing evidence on the use of LLMs for title/abstract screening, citation filtering, and study selection in medical and health-related systematic reviews. We will assess multiple performance indicators, including sensitivity, specificity, precision, recall, F1-score, workload reduction, and time efficiency. We also aim to explore sources of heterogeneity, such as model type, prompt design, clinical field, screening task complexity, dataset characteristics, and evaluation methodology. The review will follow PRISMA 2020 guidelines. Searches will be conducted across PubMed, Embase, Scopus, Web of Science, and preprint servers. Eligible studies will include evaluations of LLM-assisted or fully automated literature screening workflows, using either real-world systematic review datasets or curated benchmark sets. Risk of bias will be assessed using adapted tools for AI performance studies. Where feasible, meta-analysis will be performed to synthesize quantitative performance metrics." +W4392791588,,10.1002/jrsm.1715,2024,Can large language models replace humans in systematic reviews? Evaluating GPT ‐4's efficacy in screening and extracting data from peer‐reviewed and grey literature in multiple languages,Qusai Khraisha,5,Research Synthesis Methods,266,"Systematic reviews are vital for guiding practice, research and policy, although they are often slow and labour-intensive. Large language models (LLMs) could speed up and automate systematic reviews, but their performance in such tasks has yet to be comprehensively evaluated against humans, and no study has tested Generative Pre-Trained Transformer (GPT)-4, the biggest LLM so far. This pre-registered study uses a ""human-out-of-the-loop"" approach to evaluate GPT-4's capability in title/abstract screening, full-text review and data extraction across various literature types and languages. Although GPT-4 had accuracy on par with human performance in some tasks, results were skewed by chance agreement and dataset imbalance. Adjusting for these caused performance scores to drop across all stages: for data extraction, performance was moderate, and for screening, it ranged from none in highly balanced literature datasets (~1:1) to moderate in those datasets where the ratio of inclusion to exclusion in studies was imbalanced (~1:3). When screening full-text literature using highly reliable prompts, GPT-4's performance was more robust, reaching ""human-like"" levels. Although our findings indicate that, currently, substantial caution should be exercised if LLMs are being used to conduct systematic reviews, they also offer preliminary evidence that, for certain review tasks delivered under specific conditions, LLMs can rival human performance." +W7138880922,,10.64898/2026.03.17.26348656,2026,Performance of Large Language Models in Automated Medical Literature Screening: A Systematic Review and Meta-analysis,Xie Chenggong,7,medRxiv,0,"Abstract Objective To systematically evaluate the diagnostic performance of large language models (LLMs) in automated medical literature screening and to determine their potential role in supporting evidence synthesis workflows. Methods A systematic review and meta-analysis was conducted according to PRISMA DTA guidance. PubMed, Web of Science, Embase, the Cochrane Library and Google Scholar were searched from 1 January 2022 to 17 November 2025. Studies assessing LLMs for automated title and abstract screening or full-text eligibility assessment in medical literature were included. Diagnostic accuracy metrics were extracted and pooled using a bivariate random effects model and hierarchical summary receiver operating characteristic (HSROC) analysis. Subgroup analyses and meta-regression were performed to explore sources of heterogeneity. Results Eighteen studies published between 2023 and 2025 were included. In title and abstract screening, the pooled sensitivity was 0.92 and pooled specificity was 0.94. The SROC area under the curve (AUC) reached 0.98. In full-text screening, pooled sensitivity and specificity both reached 0.99 and the AUC was 0.99. Prompt strategies incorporating examples or chain-of-thought reasoning significantly improved sensitivity. Across studies, most models were deployed without task specific fine tuning and still achieved strong performance. Subgroup analyses and meta regression did not identify significant sources of heterogeneity. Many studies also reported substantial efficiency gains, including large reductions in screening workload, time and cost. Conclusion LLMs demonstrate high diagnostic accuracy for automated medical literature screening, particularly in full-text assessment. These models show strong potential as high sensitivity assistive tools that can substantially reduce manual screening burden while supporting evidence synthesis. Further methodological optimization and validation in large scale real-world settings are required to establish their long term role in evidence-based medicine." +W7171023253,,10.1111/jebm.70166,2026,Performance of Large Language Models in Automated Medical Literature Screening: A Systematic Review and Meta‐Analysis,Chenggong Xie,7,Journal of Evidence-Based Medicine,0,"OBJECTIVE: To systematically evaluate the diagnostic performance of large language models (LLMs) in automated medical literature screening and to determine their potential role in supporting evidence synthesis workflows. METHODS: PubMed, Web of Science, Embase, The Cochrane Library, Google Scholar, CNKI, Wanfang, VIP, and CBM were searched from January 1, 2022 to June 11, 2026. Studies assessing LLMs for automated title and abstract screening or full-text eligibility assessment in medical literature were included. The primary outcomes were sensitivity and specificity. Secondary outcomes included positive and negative likelihood ratios, diagnostic odds ratio, area under the curve (AUC), and efficiency-related metrics. Pooled sensitivity and specificity were estimated using a bivariate random-effects model and hierarchical summary receiver operating characteristic framework. Subgroup analyses and meta-regression were performed to explore sources of heterogeneity. This systematic review was registered with the Open Science Framework (https://osf.io/56d8j). RESULTS: Eighteen studies published between 2023 and 2025 were included. In title and abstract screening, the pooled sensitivity was 0.92 (95% confidence interval [CI]: 0.81-0.96) and pooled specificity was 0.94 (95% CI: 0.90-0.97). The summary receiver operating characteristic AUC reached 0.98 (95% CI: 0.96-0.99). In full-text screening, pooled sensitivity and specificity both reached 0.99 (95% CI: 0.95-1.00) and the AUC was 0.99 (95% CI: 0.98-1.00). Prompt strategies incorporating examples or chain-of-thought reasoning were associated with higher sensitivity than strategies without these approaches (0.95 vs. 0.86, p < 0.01). Several studies reported substantial efficiency gains, including workload reductions ranging from approximately 50% to 99% and screening time reductions of up to tenfold. CONCLUSIONS: LLMs shows promising performance in automated medical literature screening, particularly in full-text assessment. These models show strong potential as high sensitivity assistive tools that can substantially reduce manual screening burden while supporting evidence synthesis. Further real-world validation is needed to establish their role in evidence-based medicine." +W4390127073,,10.1016/j.jval.2023.09.030,2023,P21 A Comparative Analysis of Large Language Models (LLM) Utilised in Systematic Literature Review,H Rathi,4,Value in Health,4, +W4410000915,,10.3390/info16050378,2025,"Large Language Models in Systematic Review Screening: Opportunities, Challenges, and Methodological Considerations",Carlo Galli,3,Information,35,"Large Language Models (LLMs) have recently emerged as a powerful option for partially automating the labor-intensive process of screening articles in systematic reviews. Unlike traditional semi-automated platforms that rely on iterative human feedback, LLM-based pipelines can operate in a zero-shot or few-shot manner, classifying abstracts according to predefined criteria. This paper offers a step-by-step methodology for researchers, librarians, and students seeking to incorporate LLMs—such as GPT-4—into systematic reviews. It discusses required software and data preprocessing, presents various prompt strategies, and emphasizes the importance of human oversight to maintain rigorous quality control. The proposed framework aims to provide best practices and offer guidance on managing costs, reproducibility, and prompt refinement. By following these guidelines, review teams can substantially reduce screening workloads without compromising the comprehensive nature of evidence-based research." +W4400099044,,10.3389/fdsfr.2024.1379260,2024,Assessing the performance of large language models in literature screening for pharmacovigilance: a comparative study,Dan Li,8,Frontiers in Drug Safety and Regulation,18,"Pharmacovigilance plays a crucial role in ensuring the safety of pharmaceutical products. It involves the systematic monitoring of adverse events and the detection of potential safety concerns related to drugs. Manual literature screening for pharmacovigilance related articles is a labor-intensive and time-consuming task, requiring streamlined solutions to cope with the continuous growth of literature. The primary objective of this study is to assess the performance of Large Language Models (LLMs) in automating literature screening for pharmacovigilance, aiming to enhance the process by identifying relevant articles more effectively. This study represents a novel application of LLMs including OpenAI's GPT-3.5, GPT-4, and Anthropic's Claude2, in the field of pharmacovigilance, evaluating their ability to categorize medical publications as relevant or irrelevant for safety signal reviews. Our analysis encompassed N-shot learning, chain-of-thought reasoning, and evaluating metrics, with a focus on factors impacting accuracy. The findings highlight the promising potential of LLMs in literature screening, achieving a reproducibility of 93%, sensitivity of 97%, and specificity of 67% showcasing notable strengths in terms of reproducibility and sensitivity, although with moderate specificity. Notably, performance improved when models were provided examples consisting of abstracts, labels, and corresponding reasoning explanations. Moreover, our exploration identified several potential contributing factors influencing prediction outcomes. These factors encompassed the choice of key words and prompts, the balance of the examples, and variations in reasoning explanations. By configuring advanced LLMs for efficient screening of extensive literature databases, this study underscores the transformative potential of these models in drug safety monitoring. Furthermore, these insights gained from this study can inform the development of automated systems for pharmacovigilance, contributing to the ongoing efforts to ensure the safety and efficacy of pharmacovigilance products." +W4411040734,,10.1016/j.jdent.2025.105877,2025,Large language models for the screening step in systematic reviews in dentistry,Rata Rokhshad,5,Journal of Dentistry,8, +W4395052272,,10.1016/j.isci.2024.109713,2024,The application of large language models in medicine: A scoping review,Xiangbin Meng,25,iScience,414,"This study systematically reviewed the application of large language models (LLMs) in medicine, analyzing 550 selected studies from a vast literature search. LLMs like ChatGPT transformed healthcare by enhancing diagnostics, medical writing, education, and project management. They assisted in drafting medical documents, creating training simulations, and streamlining research processes. Despite their growing utility in assisted diagnosis and improving doctor-patient communication, challenges persisted, including limitations in contextual understanding and the risk of over-reliance. The surge in LLM-related research indicated a focus on medical writing, diagnostics, and patient communication, but highlighted the need for careful integration, considering validation, ethical concerns, and the balance with traditional medical practice. Future research directions suggested a focus on multimodal LLMs, deeper algorithmic understanding, and ensuring responsible, effective use in healthcare." +W4415594129,,10.2196/76947,2025,Evaluating Large Language Models in Ophthalmology: Systematic Review,Zili Zhang,7,Journal of Medical Internet Research,34,"BACKGROUND Large language models (LLMs) have the potential to revolutionize ophthalmic care, but their evaluation practice remains fragmented. A systematic assessment is crucial to identify gaps and guide future evaluation practices and clinical integration. OBJECTIVE This study aims to map the current landscape of LLM evaluations in ophthalmology and explore whether performance synthesis is feasible for a common task. METHODS A comprehensive search of PubMed, Web of Science, Embase, and IEEE Xplore was conducted up to November 17, 2024 (no language limits). Eligible publications quantitatively assessed an existing or modified LLM on ophthalmology-related tasks. Studies without full-text availability or those focusing solely on vision-only models were excluded. Two reviewers screened studies and extracted data across 6 dimensions (evaluated LLM, data modality, ophthalmic subspecialty, medical task, evaluation dimension, and clinical alignment), and disagreements were resolved by a third reviewer. Descriptive statistics were analyzed and visualized using Python (with NumPy, Pandas, SciPy, and Matplotlib libraries). The Fisher exact test compared open- versus closed-source models. An exploratory random-effects meta-analysis (logit transformation; DerSimonian-Laird τ2) was performed for the diagnosis-making task; heterogeneity was reported with I2 and subgrouped by model, modality, and subspecialty. RESULTS Of the 817 identified records, 187 studies met the inclusion criteria. Closed-source LLMs dominated: 170 for ChatGPT, 58 for Gemini, and 32 for Copilot. Open-source LLMs appeared in only 25 (13.4%) of studies overall, but they appeared in 17 (77.3%) of evaluation-after-development studies, versus 8 (4.8%) pure-evaluation studies (P<1×10-5). Evaluations were chiefly text-only (n=168); image-text tasks, despite the centrality of imaging, were used in 19 studies. Subspecialty coverage was skewed toward comprehensive ophthalmology (n=72), retina and vitreous (n=32), and glaucoma (n=20). Refractive surgery, ocular pathology and oncology, and ophthalmic pharmacology each appeared in 3 or fewer studies. Medical query (n=86), standardized examination (n=41), and diagnosis making (n=29) emerged as the 3 predominant tasks, while research assistance (n=5), patient triaging (n=3), and disease prediction (n=3) received less attention. Accuracy was reported in most studies (n=176), whereas calibration and uncertainty were almost absent (n=5). Real-world patient data (n=45), human performance comparison (n=63), non‑English testing (n=24), and real-world deployment (n=4) were relatively absent. Exploratory meta-analysis pooled 28 diagnostic evaluations from 17 studies: overall accuracy was 0.594 (95% CI 0.488-0.692) with extreme heterogeneity (I2=94.5%). Subgroups remained heterogeneous (I2>80%), and findings were inconsistent (eg, pooled GPT-3.5>GPT-4). CONCLUSIONS Evidence on LLM evaluations in ophthalmology is extensive but heterogeneous. Most studies have tested a few closed-source LLMs on text-based questions, leaving open-source systems, multimodal tasks, non-English contexts, and real-world deployment underexamined. High methodological variability precludes meaningful performance aggregation, as illustrated by the heterogeneous meta-analysis. Standardized, multimodal benchmarks and phased clinical validation pipelines are urgently needed before LLMs can be safely integrated into eye care workflows." +W7129376919,,10.64898/2026.02.07.26345640,2026,Collaborative large language models (LLMs) are all you need for screening in systematic reviews,Mihir Parmar,16,medRxiv,1,"Background: The ability of large language models (LLMs) to work collaboratively and screen studies in a systematic review (SR) is under-explored. Hence, we aimed to evaluate the effectiveness of LLMs in automating the process of screening in systematic reviews. Methods: This is an observational study which included labeled data (title and abstracts) for five SRs. Originally, two reviewers screened the citations independently for eligibility. A third reviewer cross-checked each citation for quality assurance. GPT-4, Claude-3-Sonnet, and Gemini-Pro-1.0 were used using zero-shot chain-of-thought prompting. Collaborative approaches included (i): conflict resolution using benefit of the doubt, (ii) majority voting using an independent third LLM and (iii) conflict resolution using an informed third LLM. Performance was assessed using accuracy, precision for exclusion, and recall for inclusion. Work saved over samples (WSS) was computed to estimate the reduction in manual human effort. Results: A total of 11300 articles were included in this study. The individual models, GPT-4, Claude-3-Sonnet, and Gemini-Pro-1.0 exhibited a high precision for exclusion, achieving 99.7%, 99.7%, and 99.2% and high recall for inclusion achieving 95.5%, 96.6% and 85.7%, respectively. However, the collaborative approach utilizing the two best-performing models (GPT-4 and Claude-3S) achieved an average precision of 99.9% and a recall of 98.5% (across all collaborative approaches). Furthermore, the proposed collaborative approach resulted in an average WSS of 63.5%, compared to the average WSS of 45.2% for individual models. Conversational LLM interactions showed a consistent pattern of results. Limitations: This study was limited due to reliance on proprietary models, and evaluation on oncology datasets. Conclusion: Evidence shows that collaborative LLMs enable efficient, high-performing screening in systematic reviews, supporting continuous evidence updates. Primary funding source: NIH (U24CA265879-01-1) and Carolyn-Ann-Kennedy-Bacon Fund." +W4408837107,,10.1016/j.bjoms.2025.03.006,2025,The impact of the large language model ChatGPT in oral and maxillofacial surgery: a systematic review,Lucas Mariz de Menezes Torres,5,British Journal of Oral and Maxillofacial Surgery,12, +W7213443824,,10.1108/ajim-06-2025-0363,2026,Large language models for literature screening in conceptually complex and interdisciplinary reviews,Ni Cheng,5,Aslib Journal of Information Management,0,"Purpose This study examines the capabilities and limitations of large language models (LLMs) for literature screening in systematic reviews involving conceptually diffuse and interdisciplinary topics. Design/methodology/approach Two screening datasets were constructed. Four LLMs, Claude 4.6, DeepSeek V4 Pro, Gemini 3.1 Pro, and GPT-5.5, were evaluated using a staged, rule-guided prompting workflow covering title screening, abstract screening, and full-text assessment. Model performance was assessed using accuracy, precision, recall, specificity, the F1-score, and Cohen's kappa, supplemented by stage-wise screening flow analysis and qualitative discrepancy analysis between model and human reviewer decisions. Findings Model performance varied substantially across datasets and models. In the conceptually diffuse Dataset A, all models achieved high specificity, but positive-class performance was more limited. DeepSeek V4 Pro achieved the highest accuracy, precision, F1-score, and Cohen's kappa, whereas Claude 4.6 achieved the highest recall. In the more clearly bounded Dataset B, recall was higher and model behavior was more convergent, with GPT-5.5 showing the best overall balance across performance indicators. Stage-wise analysis showed that models differed in where they made inclusion and exclusion decisions, while discrepancy analysis indicated that errors were mainly related to conceptual scope confusion, context misidentification, and underestimation of analytical depth. These findings suggest that current LLMs can support literature screening, but their reliability depends strongly on the conceptual clarity of the review task and the structure of eligibility criteria. Originality/value This study extends the literature on LLM-assisted screening by focusing on conceptually abstract and interdisciplinary review tasks. This study introduces a staged, Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA)-aligned screening workflow and conducts an error analysis that explains how LLM screening can fail in semantically diffuse settings. The study provides practical implications for the transparent and responsible use of LLMs in informetric research and systematic review practice." +W4407873703,,10.7326/annals-25-00012,2025,Large Language Models for Screening Search Results in Systematic Reviews: Are We There Yet?,S. Swaroop Vedula,2,Annals of Internal Medicine,1, +W4399210824,,10.2196/56780,2024,Potential Roles of Large Language Models in the Production of Systematic Reviews and Meta-Analyses,Xufei Luo,10,Journal of Medical Internet Research,76,"Large language models (LLMs) such as ChatGPT have become widely applied in the field of medical research. In the process of conducting systematic reviews, similar tools can be used to expedite various steps, including defining clinical questions, performing the literature search, document screening, information extraction, and language refinement, thereby conserving resources and enhancing efficiency. However, when using LLMs, attention should be paid to transparent reporting, distinguishing between genuine and false content, and avoiding academic misconduct. In this viewpoint, we highlight the potential roles of LLMs in the creation of systematic reviews and meta-analyses, elucidating their advantages, limitations, and future research directions, aiming to provide insights and guidance for authors planning systematic reviews and meta-analyses." +W4401759841,,10.1186/s13643-024-02609-x,2024,Evaluating the effectiveness of large language models in abstract screening: a comparative analysis,Michael Li,3,Systematic Reviews,127,"OBJECTIVE: This study aimed to evaluate the performance of large language models (LLMs) in the task of abstract screening in systematic review and meta-analysis studies, exploring their effectiveness, efficiency, and potential integration into existing human expert-based workflows. METHODS: We developed automation scripts in Python to interact with the APIs of several LLM tools, including ChatGPT v4.0, ChatGPT v3.5, Google PaLM 2, and Meta Llama 2, and latest tools including ChatGPT v4.0 turbo, ChatGPT v3.5 turbo, Google Gemini 1.0 pro, Meta Llama 3, and Claude 3. This study focused on three databases of abstracts and used them as benchmarks to evaluate the performance of these LLM tools in terms of sensitivity, specificity, and overall accuracy. The results of the LLM tools were compared to human-curated inclusion decisions, gold standard for systematic review and meta-analysis studies. RESULTS: Different LLM tools had varying abilities in abstract screening. Chat GPT v4.0 demonstrated remarkable performance, with balanced sensitivity and specificity, and overall accuracy consistently reaching or exceeding 90%, indicating a high potential for LLMs in abstract screening tasks. The study found that LLMs could provide reliable results with minimal human effort and thus serve as a cost-effective and efficient alternative to traditional abstract screening methods. CONCLUSION: While LLM tools are not yet ready to completely replace human experts in abstract screening, they show great promise in revolutionizing the process. They can serve as autonomous AI reviewers, contribute to collaborative workflows with human experts, and integrate with hybrid approaches to develop custom tools for increased efficiency. As technology continues to advance, LLMs are poised to play an increasingly important role in abstract screening, reshaping the workflow of systematic review and meta-analysis studies." +W4410178347,,10.1186/s12929-025-01131-z,2025,Impact of large language model (ChatGPT) in healthcare: an umbrella review and evidence synthesis,Usman Iqbal,6,Journal of Biomedical Science,133,"BACKGROUND: The emergence of Artificial Intelligence (AI), particularly Chat Generative Pre-Trained Transformer (ChatGPT), a Large Language Model (LLM), in healthcare promises to reshape patient care, clinical decision-making, and medical education. This review aims to synthesise research findings to consolidate the implications of ChatGPT integration in healthcare and identify research gaps. MAIN BODY: The umbrella review was conducted following Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) guidelines. The Cochrane Library, PubMed, Scopus, Web of Science, and Google Scholar were searched from inception until February 2024. Due to the heterogeneity of the included studies, no quantitative analysis was performed. Instead, information was extracted, summarised, synthesised, and presented in a narrative form. Two reviewers undertook title, abstract, and full text screening independently. The methodological quality and overall rating of the included reviews were assessed using the A Measurement Tool to Assess systematic Reviews (AMSTAR-2) checklist. The review examined 17 studies, comprising 15 systematic reviews and 2 meta-analyses, on ChatGPT in healthcare, revealing diverse focuses. The AMSTAR-2 assessment identified 5 moderate and 12 low-quality reviews, with deficiencies like study design justification and funding source reporting. The most reported theme that emerged was ChatGPT's use in disease diagnosis or clinical decision-making. While 82.4% of studies focused on its general usage, 17.6% explored unique topics like its role in medical examinations and conducting systematic reviews. Among these, 52.9% targeted general healthcare, with 41.2% focusing on specific domains like radiology, neurosurgery, gastroenterology, public health dentistry, and ophthalmology. ChatGPT's use for manuscript review or writing was mentioned in 17.6% of reviews. Promising applications include enhancing patient care and clinical decision-making, though ethical, legal, and accuracy concerns require cautious integration. CONCLUSION: We summarise the identified areas in reviews regarding ChatGPT's transformative impact in healthcare, highlighting patient care, decision-making, and medical education. Emphasising the importance of ethical regulations and the involvement of policymakers, we urge further investigation to ensure the reliability of ChatGPT and to promote trust in healthcare and research." +W4392938433,,10.1007/s00432-024-05678-6,2024,Utilizing large language models in breast cancer management: systematic review,Vera Sorin,7,Journal of Cancer Research and Clinical Oncology,45,"PURPOSE: Despite advanced technologies in breast cancer management, challenges remain in efficiently interpreting vast clinical data for patient-specific insights. We reviewed the literature on how large language models (LLMs) such as ChatGPT might offer solutions in this field. METHODS: We searched MEDLINE for relevant studies published before December 22, 2023. Keywords included: ""large language models"", ""LLM"", ""GPT"", ""ChatGPT"", ""OpenAI"", and ""breast"". The risk bias was evaluated using the QUADAS-2 tool. RESULTS: Six studies evaluating either ChatGPT-3.5 or GPT-4, met our inclusion criteria. They explored clinical notes analysis, guideline-based question-answering, and patient management recommendations. Accuracy varied between studies, ranging from 50 to 98%. Higher accuracy was seen in structured tasks like information retrieval. Half of the studies used real patient data, adding practical clinical value. Challenges included inconsistent accuracy, dependency on the way questions are posed (prompt-dependency), and in some cases, missing critical clinical information. CONCLUSION: LLMs hold potential in breast cancer care, especially in textual information extraction and guideline-driven clinical question-answering. Yet, their inconsistent accuracy underscores the need for careful validation of these models, and the importance of ongoing supervision." +W4411329844,,10.1007/s12630-025-02973-9,2025,The applications of ChatGPT and other large language models in anesthesiology and critical care: a systematic review,Nicolas Daccache,4,Canadian Journal of Anesthesia/Journal canadien d anesthésie,15, +W4399707508,,10.1016/j.jval.2024.03.1461,2024,MSR28 Use of Large Language Model (LLM) for Full-Text Screening in Systematic Literature Reviews: A Comparative Analysis,H Rathi,4,Value in Health,2, +W4392964628,,10.1007/978-3-031-56027-9_25,2024,Zero-Shot Generative Large Language Models for Systematic Review Screening Automation,Shuai Wang,6,Lecture notes in computer science,23, +W4409341650,,10.1016/j.future.2025.107829,2025,The role of Large Language Models in addressing IoT challenges: A systematic literature review,Gabriele De Vito,3,Future Generation Computer Systems,20,"The Internet of Things (IoT) has revolutionized various sectors by enabling devices to communicate and interact seamlessly. However, developing IoT applications has data management, security, and interoperability challenges. Large Language Models (LLMs) have shown promise in addressing these challenges due to their advanced language processing capabilities. This Systematic Literature Review assesses the role of LLMs in addressing IoT challenges, exploring the strategies, hardware, and software configurations used, and identifying directions for future research. We extensively searched databases like Scopus, IEEE Xplore, and ACM Digital Library, initially screening 1419 studies and identifying an additional 1167 through snowballing, ultimately focusing on 55 relevant papers. The findings reveal LLMs’ potential to address key IoT challenges such as security and scalability. However, they also highlight significant obstacles, including high computational demands and the complexities of training and tuning these models. Future research should aim to develop methods to reduce the computational requirements of LLMs, improve training datasets, simplify implementation processes, and explore the ethical and privacy implications of using LLMs in IoT applications." +W4402141055,,10.1101/2024.09.01.24312887,2024,Large Language Models in Cardiology: A Systematic Review,Moran Gendler,8,medRxiv,5,"Background: Large language models (LLMs) are increasingly used in health care, but their role in cardiology has not yet been systematically evaluated. Objective: This review aimed to assess the applications, performance, and limitations of LLMs across diverse cardiology tasks, including chronic and progressive conditions, acute events, education, and diagnostic testing. Methods: A systematic search was conducted in PubMed and Scopus for studies published up to April 14, 2024, using keywords related to LLMs and cardiology. Studies evaluating LLM outputs in cardiology-related tasks were included. Data were extracted across 5 predefined domains and the risk of bias was assessed using an adapted QUADAS-2 tool (developed by Whiting et al at the University of Bristol). The review protocol was registered in PROSPERO (CRD42024556397). Results: A total of 33 studies contributed quantitative outcome data to a descriptive synthesis. Across chronic conditions, ChatGPT-3.5 (OpenAI) answered 91% (43/47) heart failure questions accurately, although readability often required college-level comprehension. In acute scenarios, Bing Chat omitted key myocardial infarction first aid steps in 25% (5/20) to 45% (9/20) of cases, while cardiac arrest information was rated highly (mean 4.3/5, SD 0.7) but written above recommended reading levels. In physician education tasks, ChatGPT-4 (OpenAI) demonstrated higher accuracy than ChatGPT-3.5, improving from 38% (33/88) to 66% (58/88). In patient education studies, ChatGPT-4 provided scientifically adequate explanations (5.0-6.0/7) comparable to hospital materials but at higher reading levels (11th vs 7th grade). In diagnostic testing, ChatGPT-4 interpreted 91% (36/40) electrocardiogram vignettes correctly, significantly better than emergency physicians (31/40, 77%; P< .001), but showed lower performance in echocardiography. Conclusions: LLMs show meaningful potential in cardiology, especially for education and electrocardiogram interpretation, but performance varies across clinical tasks. Limitations in emergency guidance and readability, as well as small in silico study designs, highlight the need for multimodal models and prospective validation." +W4391561379,,10.1038/s42256-023-00788-1,2024,Leveraging large language models for predictive chemistry,Kevin Maik Jablonka,4,Nature Machine Intelligence,377,"Abstract Machine learning has transformed many fields and has recently found applications in chemistry and materials science. The small datasets commonly found in chemistry sparked the development of sophisticated machine learning approaches that incorporate chemical knowledge for each application and, therefore, require specialized expertise to develop. Here we show that GPT-3, a large language model trained on vast amounts of text extracted from the Internet, can easily be adapted to solve various tasks in chemistry and materials science by fine-tuning it to answer chemical questions in natural language with the correct answer. We compared this approach with dedicated machine learning models for many applications spanning the properties of molecules and materials to the yield of chemical reactions. Surprisingly, our fine-tuned version of GPT-3 can perform comparably to or even outperform conventional machine learning techniques, in particular in the low-data limit. In addition, we can perform inverse design by simply inverting the questions. The ease of use and high performance, especially for small datasets, can impact the fundamental approach to using machine learning in the chemical and material sciences. In addition to a literature search, querying a pre-trained large language model might become a routine way to bootstrap a project by leveraging the collective knowledge encoded in these foundation models, or to provide a baseline for predictive tasks." +W7141008388,10.48550/arxiv.2603.24767,10.48550/arxiv.2603.24767,2026,Fine-Tuning A Large Language Model for Systematic Review Screening,Kweku Andoh Yamoah,5,arXiv (Cornell University),0,"Systematic reviews traditionally have taken considerable amounts of human time and energy to complete, in part due to the extensive number of titles and abstracts that must be reviewed for potential inclusion. Recently, researchers have begun to explore how to use large language models (LLMs) to make this process more efficient. However, research to date has shown inconsistent results. We posit this is because prompting alone may not provide sufficient context for the model(s) to perform well. In this study, we fine-tune a small 1.2 billion parameter open-weight LLM specifically for study screening in the context of a systematic review in which humans rated more than 8500 titles and abstracts for potential inclusion. Our results showed strong performance improvements from the fine-tuned model, with the weighted F1 score improving 80.79% compared to the base model. When run on the full dataset of 8,277 studies, the fine-tuned model had 86.40% agreement with the human coder, a 91.18% true positive rate, a 86.38% true negative rate, and perfect agreement across multiple inference runs. Taken together, our results show that there is promise for fine-tuning LLMs for title and abstract screening in large-scale systematic reviews." +W4404651681,,10.1007/s00701-024-06372-9,2024,Large language models in neurosurgery: a systematic review and meta-analysis,Advait Patil,6,Acta Neurochirurgica,26, +W7161451900,,,2026,Large Language Models for Web Accessibility: A Systematic Literature Review,Wajdi Aljedaani,2,arXiv (Cornell University),0,"Web accessibility aims to ensure that web content and services are usable by people with diverse abilities. In recent years, Large Language Models (LLMs) have been increasingly explored to support accessibility-related tasks on the web, such as content generation, issue detection, and remediation. However, little is known about the characteristics of these approaches, the accessibility issues they target, the standards they follow, and how they are evaluated. In this paper, we present a systematic literature review of 38 peer-reviewed studies that investigate the use of LLMs in web accessibility contexts. We begin by performing a comprehensive search of scientific publications to identify relevant studies. We then conduct a comparative analysis to examine the accessibility tasks addressed, the LLM models and prompting strategies employed, the system architectures adopted, the accessibility issues and guidelines considered, and the evaluation methods used across studies. Our findings show that most studies apply LLMs to text-centric and structurally explicit accessibility tasks, with WCAG serving as the primary reference framework and limited consideration of cognitive accessibility guidelines (COGA). The reviewed approaches predominantly rely on general-purpose LLMs and prompt-based interactions, while evaluation practices vary widely and often lack direct involvement of users with disabilities. We envision this review as a consolidated reference for researchers and practitioners seeking to understand the current landscape of LLM-supported web accessibility, and as a foundation to guide future research and tool development in this area." +W7142557847,,,2026,Fine-Tuning A Large Language Model for Systematic Review Screening,Kweku Andoh Yamoah,5,arXiv (Cornell University),0,"Systematic reviews traditionally have taken considerable amounts of human time and energy to complete, in part due to the extensive number of titles and abstracts that must be reviewed for potential inclusion. Recently, researchers have begun to explore how to use large language models (LLMs) to make this process more efficient. However, research to date has shown inconsistent results. We posit this is because prompting alone may not provide sufficient context for the model(s) to perform well. In this study, we fine-tune a small 1.2 billion parameter open-weight LLM specifically for study screening in the context of a systematic review in which humans rated more than 8500 titles and abstracts for potential inclusion. Our results showed strong performance improvements from the fine-tuned model, with the weighted F1 score improving 80.79% compared to the base model. When run on the full dataset of 8,277 studies, the fine-tuned model had 86.40% agreement with the human coder, a 91.18% true positive rate, a 86.38% true negative rate, and perfect agreement across multiple inference runs. Taken together, our results show that there is promise for fine-tuning LLMs for title and abstract screening in large-scale systematic reviews." +W7161293256,10.48550/arxiv.2605.13873,10.48550/arxiv.2605.13873,2026,Large Language Models for Web Accessibility: A Systematic Literature Review,Wajdi Aljedaani,2,arXiv (Cornell University),0,"Web accessibility aims to ensure that web content and services are usable by people with diverse abilities. In recent years, Large Language Models (LLMs) have been increasingly explored to support accessibility-related tasks on the web, such as content generation, issue detection, and remediation. However, little is known about the characteristics of these approaches, the accessibility issues they target, the standards they follow, and how they are evaluated. In this paper, we present a systematic literature review of 38 peer-reviewed studies that investigate the use of LLMs in web accessibility contexts. We begin by performing a comprehensive search of scientific publications to identify relevant studies. We then conduct a comparative analysis to examine the accessibility tasks addressed, the LLM models and prompting strategies employed, the system architectures adopted, the accessibility issues and guidelines considered, and the evaluation methods used across studies. Our findings show that most studies apply LLMs to text-centric and structurally explicit accessibility tasks, with WCAG serving as the primary reference framework and limited consideration of cognitive accessibility guidelines (COGA). The reviewed approaches predominantly rely on general-purpose LLMs and prompt-based interactions, while evaluation practices vary widely and often lack direct involvement of users with disabilities. We envision this review as a consolidated reference for researchers and practitioners seeking to understand the current landscape of LLM-supported web accessibility, and as a foundation to guide future research and tool development in this area." +W4412650425,,10.1093/ehjdh/ztaf028,2025,Applications of large language models in cardiovascular disease: a systematic review,José Ferreira Santos,4,European Heart Journal - Digital Health,33,"Cardiovascular disease (CVD) remains the leading cause of morbidity and mortality worldwide. Large language models (LLMs) offer potential solutions for enhancing patient education and supporting clinical decision-making. This study aimed to evaluate LLMs' applications in CVD and explore their current implementation, from prevention to treatment. Following the Preferred Reporting Items for Systematic Reviews and Meta-Analyses guidelines, this systematic review assessed LLM applications in CVD. A comprehensive PubMed search identified relevant studies. The review prioritized pragmatic and practical applications of LLMs. Key applications, benefits, and limitations of LLMs in CVD prevention were summarized. Thirty-five observational studies met the eligibility criteria. Of these, 54% addressed primary prevention and risk factor management, while 46% focused on established CVD. Commercial LLMs were evaluated in all but one study, with 91% (32 studies) assessing ChatGPT. The LLM applications were categorized as follows: 72% addressed patient education, 17% clinical decision support, and 11% both. In 68% of studies, the primary objective was to evaluate LLMs' performance in answering frequently asked patient questions, with results indicating accurate, comprehensive, and generally safe responses. However, occasional misinformation and hallucinated references were noted. Additional applications included patient guidance on CVD, first aid, and lifestyle recommendations. Large language models were assessed for medical questions, diagnostic support, and treatment recommendations in clinical decision support. Large language models hold significant potential in CVD prevention and treatment. Evidence supports their potential as an alternative source of information for addressing patients' questions about common CVD. However, further validation is needed for their application in individualized care, from diagnosis to treatment." +W7124830196,,10.2196/76734,2026,Large Language Models in Cardiology: Systematic Review,Moran Gendler,8,JMIR Cardio,3,"Abstract Background Large language models (LLMs) are increasingly used in health care, but their role in cardiology has not yet been systematically evaluated. Objective This review aimed to assess the applications, performance, and limitations of LLMs across diverse cardiology tasks, including chronic and progressive conditions, acute events, education, and diagnostic testing. Methods A systematic search was conducted in PubMed and Scopus for studies published up to April 14, 2024, using keywords related to LLMs and cardiology. Studies evaluating LLM outputs in cardiology-related tasks were included. Data were extracted across 5 predefined domains and the risk of bias was assessed using an adapted QUADAS-2 tool (developed by Whiting et al at the University of Bristol). The review protocol was registered in PROSPERO (CRD42024556397). Results A total of 33 studies contributed quantitative outcome data to a descriptive synthesis. Across chronic conditions, ChatGPT-3.5 (OpenAI) answered 91% (43/47) heart failure questions accurately, although readability often required college-level comprehension. In acute scenarios, Bing Chat omitted key myocardial infarction first aid steps in 25% (5/20) to 45% (9/20) of cases, while cardiac arrest information was rated highly (mean 4.3/5, SD 0.7) but written above recommended reading levels. In physician education tasks, ChatGPT-4 (OpenAI) demonstrated higher accuracy than ChatGPT-3.5, improving from 38% (33/88) to 66% (58/88). In patient education studies, ChatGPT-4 provided scientifically adequate explanations (5.0‐6.0/7) comparable to hospital materials but at higher reading levels (11th vs 7th grade). In diagnostic testing, ChatGPT-4 interpreted 91% (36/40) electrocardiogram vignettes correctly, significantly better than emergency physicians (31/40, 77%; P < .001), but showed lower performance in echocardiography. Conclusions LLMs show meaningful potential in cardiology, especially for education and electrocardiogram interpretation, but performance varies across clinical tasks. Limitations in emergency guidance and readability, as well as small in silico study designs, highlight the need for multimodal models and prospective validation." +W4417475506,,10.1038/s41366-025-01992-2,2025,Large language models in obesity: a systematic review,Thanathip Suenghataiphorn,4,International Journal of Obesity,6, +W4384641573,,10.1145/3539618.3591703,2023,Can ChatGPT Write a Good Boolean Query for Systematic Review Literature Search?,Shuai Wang,4,International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR),205,"Systematic reviews are comprehensive literature reviews for a highly focused research question. These reviews are considered the highest form of evidence in medicine. Complex Boolean queries are developed as part of the systematic review creation process to retrieve literature, as they permit reproducibility and understandability. However, it is difficult and time-consuming to develop high-quality Boolean queries, often requiring the expertise of expert searchers like librarians. Recent advances in transformer-based generative models have shown their ability to effectively follow user instructions and generate answers based on these instructions. In this paper, we investigate ChatGPT as a means for automatically formulating and refining complex Boolean queries for systematic review literature search. Overall, our research finds that ChatGPT has the potential to generate effective Boolean queries. The ability of ChatGPT to follow complex instructions and generate highly precise queries makes it a tool of potential value for researchers conducting systematic reviews, particularly for rapid reviews where time is a constraint and where one can trade off higher precision for lower recall. We also identify several caveats in using ChatGPT for this task, highlighting that this technology needs further validation before it is suitable for widespread uptake." +W4399052098,,10.1111/apt.18058,2024,Systematic review: The use of large language models as medical chatbots in digestive diseases,Mauro Giuffrè,7,Alimentary Pharmacology & Therapeutics,55,"BACKGROUND: Interest in large language models (LLMs), such as OpenAI's ChatGPT, across multiple specialties has grown as a source of patient-facing medical advice and provider-facing clinical decision support. The accuracy of LLM responses for gastroenterology and hepatology-related questions is unknown. AIMS: To evaluate the accuracy and potential safety implications for LLMs for the diagnosis, management and treatment of questions related to gastroenterology and hepatology. METHODS: We conducted a systematic literature search including Cochrane Library, Google Scholar, Ovid Embase, Ovid MEDLINE, PubMed, Scopus and the Web of Science Core Collection to identify relevant articles published from inception until January 28, 2024, using a combination of keywords and controlled vocabulary for LLMs and gastroenterology or hepatology. Accuracy was defined as the percentage of entirely correct answers. RESULTS: Among the 1671 reports screened, we identified 33 full-text articles on using LLMs in gastroenterology and hepatology and included 18 in the final analysis. The accuracy of question-responding varied across different model versions. For example, accuracy ranged from 6.4% to 45.5% with ChatGPT-3.5 and was between 40% and 91.4% with ChatGPT-4. In addition, the absence of standardised methodology and reporting metrics for studies involving LLMs places all the studies at a high risk of bias and does not allow for the generalisation of single-study results. CONCLUSIONS: Current general-purpose LLMs have unacceptably low accuracy on clinical gastroenterology and hepatology tasks, which may lead to adverse patient safety events through incorrect information or triage recommendations, which might overburden healthcare systems or delay necessary care." +W4404636419,,10.1016/j.jad.2024.11.052,2024,Exploring the efficacy and potential of large language models for depression: A systematic review,Mahmud Omar,2,Journal of Affective Disorders,52, +W4415344803,,10.1145/3801961,2026,Large Language Models for Combinatorial Optimization: A Systematic Review,Francesca Da Ros,4,ACM Computing Surveys,11,"This systematic review explores the application of Large Language Models (LLMs) in Combinatorial Optimization (CO). We report our findings using the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) guidelines. We conduct a literature search via Scopus and Google Scholar, examining over 2,000 publications. We assess publications against four inclusion and four exclusion criteria related to their language, research focus, publication year, and type. Eventually, we select 103 studies. We classify these studies into semantic categories and topics to provide a comprehensive overview of the field, including the tasks performed by LLMs, the architectures of LLMs, the existing datasets specifically designed for evaluating LLMs in CO, and the field of application. Finally, we identify future directions for leveraging LLMs in this field." +W7167724220,,10.59431/jda.v5i1.810,2026,Large Language Model Integration in Desktop Applications: A Systematic Literature Review,Deyidi Mokoginta,1,Journal Dekstop Application (JDA),0,"The integration of Large Language Models (LLMs) into desktop applications has accelerated sharply over the past several years, yet no prior systematic review has examined this deployment setting as a distinct research domain — separate from web-based or mobile environments, each carrying different architectural constraints, privacy expectations, and user interaction patterns. This study maps, analyzes, and synthesizes the available scientific evidence on LLM architectures adopted in desktop implementations, the technical barriers arising during integration, the measurable effects on User Experience (UX), and the relative effectiveness of competing deployment strategies. A Systematic Literature Review (SLR) was conducted following PRISMA 2020 reporting guidelines, with research questions structured using the PICOC framework. Systematic searches across seven digital databases — IEEE Xplore, ACM Digital Library, Scopus, Web of Science, Google Scholar, arXiv, and Semantic Scholar — covering 2019–2024 yielded 1,200 initial articles, narrowed to 42 final articles through layered selection and quality appraisal using a modified CASP rubric. GPT-4 and the LLaMA family emerged as the dominant architectures, with model selection driven primarily by parameter scale and privacy requirements. Computational constraints, response latency, and data security ranked as the most serious technical barriers, with Retrieval-Augmented Generation (RAG) and model quantization as the most common mitigations. LLM integration was associated with measurable gains in task efficiency and user satisfaction in 73.8% of reviewed studies, though output predictability and user trust remained persistent challenges. A hybrid deployment approach — combining local models with cloud-based API access — emerged as the most common practice among production implementations. Unlike existing reviews addressing LLM capabilities in general software engineering contexts, this study provides the first systematic map of LLM integration specifically within the desktop application domain, offering concrete guidance for developers working at the intersection of artificial intelligence and desktop software engineering." +W4407957917,,10.1016/j.jclinepi.2025.111746,2025,"Large language models for conducting systematic reviews: on the rise, but not yet ready for use—a scoping review",Judith-Lisa Lieberum,9,Journal of Clinical Epidemiology,146,"BACKGROUND AND OBJECTIVES: Machine learning promises versatile help in the creation of systematic reviews (SRs). Recently, further developments in the form of large language models (LLMs) and their application in SR conduct attracted attention. We aimed at providing an overview of LLM applications in SR conduct in health research. METHODS: We systematically searched MEDLINE, Web of Science, IEEEXplore, ACM Digital Library, Europe PMC (preprints), Google Scholar, and conducted an additional hand search (last search: February 26, 2024). We included scientific articles in English or German, published from April 2021 onwards, building upon the results of a mapping review that has not yet identified LLM applications to support SRs. Two reviewers independently screened studies for eligibility; after piloting, 1 reviewer extracted data, checked by another. RESULTS: Our database search yielded 8054 hits, and we identified 33 articles from our hand search. We finally included 37 articles on LLM support. LLM approaches covered 10 of 13 defined SR steps, most frequently literature search (n = 15, 41%), study selection (n = 14, 38%), and data extraction (n = 11, 30%). The mostly recurring LLM was Generative Pretrained Transformer (GPT) (n = 33, 89%). Validation studies were predominant (n = 21, 57%). In half of the studies, authors evaluated LLM use as promising (n = 20, 54%), one-quarter as neutral (n = 9, 24%) and one-fifth as nonpromising (n = 8, 22%). CONCLUSION: Although LLMs show promise in supporting SR creation, fully established or validated applications are often lacking. The rapid increase in research on LLMs for evidence synthesis production highlights their growing relevance. PLAIN LANGUAGE SUMMARY: Systematic reviews are a crucial tool in health research where experts carefully collect and analyze all available evidence on a specific research question. Creating these reviews is typically time- and resource-intensive, often taking months or even years to complete, as researchers must thoroughly search, evaluate, and synthesize an immense number of scientific studies. For the present article, we conducted a review to understand how new artificial intelligence (AI) tools, specifically large language models (LLMs) like Generative Pretrained Transformer (GPT), can be used to help create systematic reviews in health research. We searched multiple scientific databases and finally found 37 relevant articles. We found that LLMs have been tested to help with various parts of the systematic review process, particularly in 3 main areas: searching scientific literature (41% of studies), selecting relevant studies (38%), and extracting important information from these studies (30%). GPT was the most commonly used LLM, appearing in 89% of the studies. Most of the research (57%) focused on testing whether these AI tools actually work as intended in this context of systematic review production. The results were mixed: about half of the studies found LLMs promising, a quarter were neutral, and one-fifth found them not promising. While LLMs show potential for making the systematic review process more efficient, there is still a lack of fully tested and validated applications. However, the increasing number of studies in this field suggests that these AI tools are becoming increasingly important in creating systematic reviews." +W4399707564,,10.1016/j.jval.2024.03.1477,2024,MSR44 How Accurate Are Large Language Models for Abstract Screening in Systematic Literature Reviews?,A Sertel,5,Value in Health,1, +W4407135263,,10.1001/jamanetworkopen.2024.57879,2025,Large Language Models for Chatbot Health Advice Studies,Bright Huo,13,JAMA Network Open,279,"Importance: There is much interest in the clinical integration of large language models (LLMs) in health care. Many studies have assessed the ability of LLMs to provide health advice, but the quality of their reporting is uncertain. Objective: To perform a systematic review to examine the reporting variability among peer-reviewed studies evaluating the performance of generative artificial intelligence (AI)-driven chatbots for summarizing evidence and providing health advice to inform the development of the Chatbot Assessment Reporting Tool (CHART). Evidence Review: A search of MEDLINE via Ovid, Embase via Elsevier, and Web of Science from inception to October 27, 2023, was conducted with the help of a health sciences librarian to yield 7752 articles. Two reviewers screened articles by title and abstract followed by full-text review to identify primary studies evaluating the clinical accuracy of generative AI-driven chatbots in providing health advice (chatbot health advice studies). Two reviewers then performed data extraction for 137 eligible studies. Findings: A total of 137 studies were included. Studies examined topics in surgery (55 [40.1%]), medicine (51 [37.2%]), and primary care (13 [9.5%]). Many studies focused on treatment (91 [66.4%]), diagnosis (60 [43.8%]), or disease prevention (29 [21.2%]). Most studies (136 [99.3%]) evaluated inaccessible, closed-source LLMs and did not provide enough information to identify the version of the LLM under evaluation. All studies lacked a sufficient description of LLM characteristics, including temperature, token length, fine-tuning availability, layers, and other details. Most studies (136 [99.3%]) did not describe a prompt engineering phase in their study. The date of LLM querying was reported in 54 (39.4%) studies. Most studies (89 [65.0%]) used subjective means to define the successful performance of the chatbot, while less than one-third addressed the ethical, regulatory, and patient safety implications of the clinical integration of LLMs. Conclusions and Relevance: In this systematic review of 137 chatbot health advice studies, the reporting quality was heterogeneous and may inform the development of the CHART reporting standards. Ethical, regulatory, and patient safety considerations are crucial as interest grows in the clinical integration of LLMs." +W4412834208,,10.1016/j.infsof.2026.108240,2026,A systematic literature review on the use of large language models for optimization algorithms,Zoiya S. Morell,4,Information and Software Technology,1,"Context: Large language models (LLMs) are becoming popular in nearly every industry, with their problem-solving capabilities, ease of use, and interpretable results touted for complex optimization problems. However, LLMs used for optimization problems are prone to hallucinations and unstructured search techniques. Meanwhile, optimization algorithms (OA) have been used for such problems for years but are often designed for specific use cases and require domain expertise. This motivated a new combined field of LLM-modified OA.Objective: Recent studies on LLM-modified OA reveal promising results in improving optimization algorithms’ capabilities but demonstrate poor standardization, use of metrics, and repeatability. Method: In this paper, we surveyed research published from January 2017 to November 2024 to uncover how researchers are using LLMs and optimization algorithms in combination and review the quality of LLM-OA research. Following an extensive manual filtering procedure and critical analysis process, we evaluated 29 relevant papers merging LLMs and optimization algorithms to answer five research questions. Results: We found an absence of repeatability measures, statistical validity tests, and accountability for LLM hallucination in LLM-OA research. Since 68\% of papers do not provide adequate statistical analysis, 72\% lack proper metrics, and 86\% disregard hallucinations, we propose a standardized research methodology and highlight avenues for future research.Conclusions: LLMs for OAs remain largely unexplored, presenting several research opportunities." +W7116968648,,10.5335/rbca.v17i3.16310,2025,A Systematic Review about Large Language Models (LLMs) applied to Code Generation,Fabio Alecsandro Bacin,4,Revista Brasileira de Computação Aplicada,1,"Large Language Models (LLMs) for code generation represent a significant advancement in software development by boosting productivity, simplifying repetitive tasks, enabling automated testing, and promoting best practices. This paper presents a systematic literature review (SLR) of studies focused on LLMs applied to code generation. The review enhances understanding of the capabilities and limitations of these models, outlining both their benefits and challenges. The review protocol included a search on the Google Scholar database using relevant keywords related to LLMs and code generation. A total of 112 articles were initially retrieved, from which 15 were selected based on relevance and quality criteria. Of these, 8 were analyzed in depth to evaluate various approaches and outcomes, while the remaining 7 provided the theoretical foundation for the study. This work contributes to the growing body of knowledge in the field and supports future research and applications of LLMs in software engineering." +W4413110542,,10.1093/dmfr/twaf060,2025,The performance of large language models in dentomaxillofacial radiology: a systematic review,Zekai Liu,7,Dentomaxillofacial Radiology,26,"OBJECTIVES: This study aimed to systematically review the current performance of large language models (LLMs) in dento-maxillofacial radiology (DMFR). METHODS: Five electronic databases were used to identify studies that developed, fine-tuned, or evaluated LLMs for DMFR-related tasks. Data extracted included study purpose, LLM type, images/text source, applied language, dataset characteristics, input and output, performance outcomes, evaluation methods, and reference standards. Customized assessment criteria adapted from the TRIPOD-LLM reporting guideline were used to evaluate the risk-of-bias in the included studies specifically regarding the clarity of dataset origin, the robustness of performance evaluation methods, and the validity of the reference standards. RESULTS: The initial search yielded 1621 titles, and 19 studies were included. These studies investigated the use of LLMs for tasks including the production and answering of DMFR-related qualification exams and educational questions (n = 8), diagnosis and treatment recommendations (n = 7), and radiology report generation and patient communication (n = 4). LLMs demonstrated varied performance in diagnosing dental conditions, with accuracy ranging from 37% to 92.5% and expert ratings for differential diagnosis and treatment planning between 3.6 and 4.7 on a 5-point scale. For DMFR-related qualification exams and board-style questions, LLMs achieved correctness rates between 33.3% and 86.1%. Automated radiology report generation showed moderate performance with accuracy ranging from 70.4% to 81.3%. CONCLUSIONS: LLMs demonstrate promising potential in DMFR, particularly for diagnostic, educational, and report generation tasks. However, their current accuracy, completeness, and consistency remain variable. Further development, validation, and standardization are needed before LLMs can be reliably integrated as supportive tools in clinical workflows and educational settings." +W7134854403,,10.21203/rs.3.rs-9057643/v1,2026,Synthetic Participants Generated by Large Language Models: A Systematic Literature Review,Eduard Kuric,3,Research Square,2, +W7125646590,,10.5281/zenodo.18370396,2026,A Systematic Literature Review on the Use of Large Language Models for Optimization Algorithms,Zoiya S. Morell,4,Zenodo (CERN European Organization for Nuclear Research),0,"Context: Large language models (LLMs) are becoming popular in nearly every industry, with their problem-solving capabilities, ease of use, and interpretable results touted for complex optimization problems. However, LLMs used for optimization problems are prone to hallucinations and unstructured search techniques. Meanwhile, optimization algorithms (OA) have been used for such problems for years but are often designed for specific use cases and require domain expertise. This motivated a new combined field of LLM-modified OA. Objective: Recent studies on LLM-modified OA reveal promising results in improving optimization algorithms’ capabilities but demonstrate poor standardization, use of metrics, and repeatability. Method: In this paper, we surveyed research published from January 2017 to November 2024 to uncover how researchers are using LLMs and optimization algorithms in combination and review the quality of LLM-OA research. Following an extensive manual filtering procedure and critical analysis process, we evaluated 41 relevant papers merging LLMs and optimization algorithms to answer five research questions. Results: Although, the majority (62.2\%) of studies use LLMs during the search process (in-the-loop), we also found an absence of repeatability measures, statistical validity tests, and accountability for LLM hallucination in LLM-OA research. Since \pct{29} of papers do not provide adequate statistical analysis and \pct{30} disregard hallucinations, we propose a standardized research methodology and highlight avenues for future research. Conclusions: LLMs for OAs remain largely unexplored, presenting several research opportunities." +W4414751697,,10.2196/74177,2025,Large Language Models in Lung Cancer: Systematic Review,Ruikang Zhong,9,Journal of Medical Internet Research,9,"Background: In the era of data and intelligence, artificial intelligence has been widely applied in the medical field. As the most cutting-edge technology, the large language model (LLM) has gained popularity due to its extraordinary ability to handle complex tasks and interactive features. Objective: This study aimed to systematically review current applications of LLMs in lung cancer (LC) care and evaluate their potential across the full-cycle management spectrum. Methods: Following PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) guidelines, we conducted a comprehensive literature search across 6 databases up to January 1, 2025. Studies were included if they satisfied the following criteria: (1) journal articles, conference papers, and preprints; (2) studies that reported the content of LLMs in LC; (3) including original data and LC-related data presented separately; and (4) studies published in English. The exclusion criteria were as follows: (1) books and book chapters, letters, reviews, conference proceedings; (2) studies that did not report the content of LLMs in LC; and (3) no original data, and LC-related data that are not presented separately. Studies were screened independently by 2 authors (SC and ZL) and assessed for quality using Quality Assessment of Diagnostic Accuracy Studies-2, Prediction Model Risk of Bias Assessment Tool, and Risk Of Bias in Non-randomized Studies - of Interventions tools, selected based on study type. Key data items extracted included model type, application scenario, prompt method, input and output format, outcome measures, and safety considerations. Data analysis was conducted using descriptive statistics. Results: Out of 706 studies screened, 28 were included (published between 2023 and 2024). The ability of LLMs to automatically extract medical records, popularize general knowledge about LC, and assist clinical diagnosis and treatment has been demonstrated through the systematic review, emerging visual ability, and multimodal potential. Prompt engineering was a critical component, with varying degrees of sophistication from zero-shot to fine-tuned approaches. Quality assessments revealed overall acceptable methodological rigor but noted limitations in bias control and data security reporting. Conclusions: LLMs show considerable potential in improving LC diagnosis, communication, and decision-making. However, their responsible use requires attention to privacy, interpretability, and human oversight." +W7171869200,,10.14744/hf.2026.65328,2026,Large Language Models in Hepatology: A Systematic Review,Thanathip Suenghataiphorn,4,Hepatology forum/Hepatology forum (Online),0,"Background and Aim: The rapid advancement of generative artificial intelligence (AI), particularly large language models (LLMs), has opened new frontiers in healthcare, with emerging implications for hepatology. This systematic review synthesizes the current state of research on the application of LLMs in hepatology, focusing on their capabilities in real-world clinical settings, limitations, and future directions. Materials and Methods: Electronic databases, including MEDLINE, EMBASE, and OVID as a search platform, were used to identify eligible studies from inception to January 2025. Eligible studies investigated the clinical utility and performance of LLMs in hepatology, with a clear comparison to a defined ground truth. Key findings were extracted and synthesized narratively. The ROBINS-I tool was used to assess the risk of bias in each study. Results: Twenty-one studies were included in this review. Our analysis reveals that LLMs demonstrate promising capabilities in processing textual and visual data related to various liver diseases, including hepatocellular carcinoma, cirrhosis, and non-alcoholic fatty liver disease. LLMs effectively assisted with radiological image interpretation, provided clinical decision support, and generated patient education materials. However, the accuracy of these models was highly variable, depending on the specific task and the complexity of the clinical scenario. Limitations, such as the generation of inaccurate or misleading information (""hallucinations""), dependence on training data quality, and ethical considerations, were identified across multiple studies. Conclusion: Generative AI demonstrates feasibility across various hepatology applications, but study heterogeneity and significant challenges remain regarding accuracy, reliability, and safety. Future integration necessitates further research into training methods, data quality, ethical considerations, and real-world validation against standardized benchmarks." +W7106340049,,10.71781/25708,2025,Automatic screening of articles for systematic reviews using large language models,"Kumar, Gauransh",1,,0,"Les revues systématiques de la littérature sont essentielles pour la recherche fondée sur des preuves mais exigent un travail manuel considérable, en particulier lors du tri des articles pour en déterminer la pertinence. Cette thèse explore comment les grands modèles de langage (Large Language Models, LLMs) peuvent aider à réduire cette charge tout en maintenant la qualité scientifique. Nous avons testé les LLMs comme assistants de sélection à laide de jeux de données variés et examiné limpact des différentes conceptions dinvite sur les performances. Notre étude a analysé la généralisabilité des invites entre différentes familles de LLMs, la relation entre la confiance du modèle et sa précision, ainsi que des stratégies pour combiner plusieurs approches. Nous avons développé PromptSLR, un outil permettant des tests contrôlés dinvites et une analyse de performance. Grâce à lélimination systématique des caractéristiques des invites, nous avons identifié des conceptions optimales à laide danalyses statistiques. Les résultats montrent que 2 invites atteignent les meilleures performances sur un ensemble de près de 30 invites, mais que leur efficacité varie fortement selon les modèles de LLM, avec une transférabilité limitée entre modèles. De façon critique, nous avons constaté que les LLMs excellent davantage à identifier les articles non pertinents quà sélectionner les pertinents, ce qui les rend idéaux pour le pré-tri plutôt que pour le tri principal. Il est observé que demander aux LLMs de sauto-évaluer (score de confiance) améliore leurs performances. Une confiance plus élevée du modèle corrèle avec une meilleure précision, permettant aux chercheurs de concentrer leffort manuel sur les cas incertains tout en automatisant lexclusion des évidences. Nous proposons une approche hybride combinant le pré-tri par LLM et le tri manuel traditionnel, ce qui réduit substantiellement la charge de travail des réviseurs tout en préservant la qualité du processus. Cette thèse positionne les LLMs comme des outils dassistance qui complètent lexpertise humaine plutôt que de la remplacer, et contribue à fournir des méthodes pratiques pour améliorer l'efficacité des SLRs ainsi que des outils pour une mise en oeuvre réelle." +W4411569746,,10.1017/rsm.2025.10014,2025,Optimal large language models to screen citations for systematic reviews,Takehiko Oami,3,Research Synthesis Methods,16,"Abstract Recent studies highlight the potential of large language models (LLMs) in citation screening for systematic reviews; however, the efficiency of individual LLMs for this application remains unclear. This study aimed to compare accuracy, time-related efficiency, cost, and consistency across four LLMs—GPT-4o, Gemini 1.5 Pro, Claude 3.5 Sonnet, and Llama 3.3 70B—for literature screening tasks. The models screened for clinical questions from the Japanese Clinical Practice Guidelines for the Management of Sepsis and Septic Shock 2024. Sensitivity and specificity were calculated for each model based on conventional citation screening results for qualitative assessment. We also recorded the time and cost of screening and assessed consistency to verify reproducibility. A post hoc analysis explored whether integrating outputs from multiple models could enhance screening accuracy. GPT-4o and Llama 3.3 70B achieved high specificity but lower sensitivity, while Gemini 1.5 Pro and Claude 3.5 Sonnet exhibited higher sensitivity at the cost of lower specificity. Citation screening times and costs varied, with GPT-4o being the fastest and Llama 3.3 70B the most cost-effective. Consistency was comparable among the models. An ensemble approach combining model outputs improved sensitivity but increased the number of false positives, requiring additional review effort. Each model demonstrated distinct strengths, effectively streamlining citation screening by saving time and reducing workload. However, reviewing false positives remains a challenge. Combining models may enhance sensitivity, indicating the potential of LLMs to optimize systematic review workflows." +W7201993829,,10.1111/jocn.70490,2026,ChatGPT and Large Language Models in Contemporary Nursing,Izabella Uchmanowicz,10,Journal of Clinical Nursing,0,"OBJECTIVES: This narrative review synthesizes published evidence on the applications, benefits, limitations and governance considerations of ChatGPT and large language models (LLMs) in nursing, across three domains: education, clinical practice and workflow management. DESIGN: The article was conducted as a narrative review. METHODS: A structured literature search was conducted in Medline (via PubMed), Scopus and arXiv, covering publications from January 2019 to March 2026. Peer-reviewed original studies, systematic reviews, scoping reviews, narrative reviews and expert commentaries addressing LLM applications in nursing education, clinical practice or workflow were eligible for inclusion. Studies limited exclusively to non-nursing medical specialties without transferable nursing implications were excluded. Findings were narratively synthesized across five thematic domains by authors with subject-matter expertise in each area. RESULTS: In nursing education, ChatGPT demonstrates utility as an adaptive cognitive scaffold, supporting theoretical learning, simulation-based training and virtual patient encounters, though unregulated use poses risks to academic integrity and independent clinical reasoning. In clinical practice, LLMs can assist with preliminary symptom assessment and patient education material generation; however, performance deteriorates markedly in complex or data-sparse clinical scenarios and hallucination rates remain clinically significant. In workflow management, ChatGPT shows promise in reducing documentation burden and supporting administrative tasks, though data privacy obligations under frameworks such as GDPR constrain real-world deployment. Across all domains, concerns persist regarding algorithmic bias, professional accountability and the absence of clear medico-legal frameworks governing AI-related clinical errors. CONCLUSION: ChatGPT and related LLMs are best positioned as auxiliary tools that augment rather than replace professional nursing judgement. Safe and ethical integration requires the development of AI literacy curricula, institutionally governed deployment frameworks, mandatory human-in-the-loop verification protocols and longitudinal evaluation of patient safety outcomes. Nurses must play an active role in shaping the responsible adoption of generative AI in healthcare. IMPLICATIONS FOR PRACTICE AND RESEARCH: 1. Nurses must treat AI-generated content as a preliminary draft requiring mandatory human verification before clinical or documentation use. 2. Institutions should prioritize closed-loop, enterprise-grade AI deployments over public platforms to ensure GDPR compliance. 3. AI literacy must be embedded in undergraduate and continuing nursing education curricula. 4. Longitudinal research on patient safety outcomes following real-world LLM deployment in nursing is urgently needed. REPORTING METHOD: As a narrative review, this article followed established guidance for the conduct and reporting of narrative reviews. PATIENT OR PUBLIC CONTRIBUTION: There was no patient or public involvement in this narrative review." +W4404288320,,10.1016/j.identj.2024.10.014,2024,Large Language Models in Dental Licensing Examinations: Systematic Review and Meta-Analysis,Mingxin Liu,7,International Dental Journal,41,"INTRODUCTION AND AIMS: This study systematically reviews and conducts a meta-analysis to evaluate the performance of various large language models (LLMs) in dental licensing examinations worldwide. The aim is to assess the accuracy of these models in different linguistic and geographical contexts. This will inform their potential application in dental education and diagnostics. METHODS: Following Preferred Reporting Items for Systematic Reviews and Meta-Analyses guidelines, we conducted a comprehensive search across PubMed, Web of Science, and Scopus for studies published from 1 January 2022 to 1 May 2024. Two authors independently reviewed the literature based on the inclusion and exclusion criteria, extracted data, and evaluated the quality of the studies in accordance with the Quality Assessment of Diagnostic Accuracy Studies-2. We conducted qualitative and quantitative analyses to evaluate the performance of LLMs. RESULTS: Eleven studies met the inclusion criteria, encompassing dental licensing examinations from eight countries. GPT-3.5, GPT-4, and Bard achieved integrated accuracy rates of 54%, 72%, and 56%, respectively. GPT-4 outperformed GPT-3.5 and Bard, passing more than half of the dental licensing examinations. Subgroup analyses and meta-regression showed that GPT-3.5 performed significantly better in English-speaking countries. GPT-4's performance, however, remained consistent across different regions. CONCLUSION: LLMs, particularly GPT-4, show potential in dental education and diagnostics, yet their accuracy remains below the threshold required for clinical application. The lack of sufficient training data in dentistry has affected LLMs' accuracy. The reliance on image-based diagnostics also presents challenges. As a result, their accuracy in dental exams is lower compared to medical licensing exams. Additionally, LLMs even provide more detailed explanation for incorrect answer than correct one. Overall, the current LLMs are not yet suitable for use in dental education and clinical diagnosis." +W4413422599,,10.1016/j.acra.2025.08.014,2025,Evaluating Large Language Models for Radiology Systematic Review Title and Abstract Screening,Siddhant Dogra,9,Academic Radiology,6, +W4414801921,,10.1016/j.jclinane.2025.112030,2025,Combining large language models enhances screening sensitivity in systematic reviews,Alessandro DE CASSAI,6,Journal of Clinical Anesthesia,3, +W4408801716,,10.1017/rsm.2025.15,2025,Validation of large language models (Llama 3 and ChatGPT-4o mini) for title and abstract screening in biomedical systematic reviews,Adriana López‐Pineda,6,Research Synthesis Methods,17,"With the increasing volume of scientific literature, there is a need to streamline the screening process for titles and abstracts in systematic reviews, reduce the workload for reviewers, and minimize errors. This study validated artificial intelligence (AI) tools, specifically Llama 3 70B via Groq's application programming interface (API) and ChatGPT-4o mini via OpenAI's API, for automating this process in biomedical research. It compared these AI tools with human reviewers using 1,081 articles after duplicate removal. Each AI model was tested in three configurations to assess sensitivity, specificity, predictive values, and likelihood ratios. The Llama 3 model's LLA_2 configuration achieved 77.5% sensitivity and 91.4% specificity, with 90.2% accuracy, a positive predictive value (PPV) of 44.3%, and a negative predictive value (NPV) of 97.9%. The ChatGPT-4o mini model's CHAT_2 configuration showed 56.2% sensitivity, 95.1% specificity, 92.0% accuracy, a PPV of 50.6%, and an NPV of 96.1%. Both models demonstrated strong specificity, with CHAT_2 having higher overall accuracy. Despite these promising results, manual validation remains necessary to address false positives and negatives, ensuring that no important studies are overlooked. This study suggests that AI can significantly enhance efficiency and accuracy in systematic reviews, potentially revolutionizing not only biomedical research but also other fields requiring extensive literature reviews." +W4408763854,,10.1093/jamia/ocaf050,2025,High-performance automated abstract screening with large language model ensembles,Rohan Sanghera,10,Journal of the American Medical Informatics Association,55,"OBJECTIVE: screening is a labor-intensive component of systematic review involving repetitive application of inclusion and exclusion criteria on a large volume of studies. We aimed to validate large language models (LLMs) used to automate abstract screening. MATERIALS AND METHODS: LLMs (GPT-3.5 Turbo, GPT-4 Turbo, GPT-4o, Llama 3 70B, Gemini 1.5 Pro, and Claude Sonnet 3.5) were trialed across 23 Cochrane Library systematic reviews to evaluate their accuracy in zero-shot binary classification for abstract screening. Initial evaluation on a balanced development dataset (n = 800) identified optimal prompting strategies, and the best performing LLM-prompt combinations were then validated on a comprehensive dataset of replicated search results (n = 119 695). RESULTS: On the development dataset, LLMs exhibited superior performance to human researchers in terms of sensitivity (LLMmax = 1.000, humanmax = 0.775), precision (LLMmax = 0.927, humanmax = 0.911), and balanced accuracy (LLMmax = 0.904, humanmax = 0.865). When evaluated on the comprehensive dataset, the best performing LLM-prompt combinations exhibited consistent sensitivity (range 0.756-1.000) but diminished precision (range 0.004-0.096) due to class imbalance. In addition, 66 LLM-human and LLM-LLM ensembles exhibited perfect sensitivity with a maximal precision of 0.458 with the development dataset, decreasing to 0.1450 over the comprehensive dataset; but conferring workload reductions ranging between 37.55% and 99.11%. DISCUSSION: Automated abstract screening can reduce the screening workload in systematic review while maintaining quality. Performance variation between reviews highlights the importance of domain-specific validation before autonomous deployment. LLM-human ensembles can achieve similar benefits while maintaining human oversight over all records. CONCLUSION: LLMs may reduce the human labor cost of systematic review with maintained or improved accuracy, thereby increasing the efficiency and quality of evidence synthesis." +W7213653222,,10.17605/osf.io/2k3zj,2026,Large language models for title and abstract screening in oncology systematic reviews,Tao Xu,2,Open Science Framework,0,"This is a retrospective registration of a completed diagnostic accuracy study. Analyses had already been conducted at the time of registration. Purpose: Title–abstract screening is a major source of workload in systematic reviews. Missing an eligible randomised trial can change the included evidence, whereas retaining an ineligible record only adds later reading. This project evaluates a predefined first-pass workflow: one frozen recall-first prompt, five commercially available large language models, and human reading limited to records labelled include or uncertain. Methods: The index test was applied to the original search yields of 45 published oncology randomised-trial meta-analyses (128,023 unique records; 611 unique reference-standard inclusions). The reference standard was each source review’s final included set after full-text assessment. Recall was scored on 641 reference-standard inclusions with abstracts that were present in all five model tables. Models were DeepSeek-chat, Qwen-Plus, Gemini 2.5 Flash, GPT-4.1 mini, and GPT-4o (temperature 0). Uncertain was counted with include. Prompt templates were compared on a locked 800-record test set after a disjoint 312-record development set; the production prompt was not changed by that catalogue. Outcomes: The primary diagnostic accuracy outcome is recall of reference-standard inclusions with abstracts. The co-primary process measure is workload reduction, 1 − (records retained for human screening / N). Secondary measures include precision, number needed to read, estimated reading time at 30 seconds per record, and the number of published inclusions that would remain unread if excluded records were not reassessed." +W4414661911,,10.3389/fdgth.2025.1659134,2025,Large language models in real-world clinical workflows: a systematic review of applications and implementation,Yaara Artsi,6,Frontiers in Digital Health,84,"Background: Large language models (LLMs) offer promise for enhancing clinical care by automating documentation, supporting decision-making, and improving communication. However, their integration into real-world healthcare workflows remains limited and under characterized. This systematic review aims to evaluate the literature on real-world implementation of LLMs in clinical workflows, including their use cases, clinical settings, observed outcomes, and challenges. Methods: We searched MEDLINE, Scopus, Web of Science, and Google Scholar for studies published between January 2015 and April 2025 that assessed LLMs in real-world clinical applications. Inclusion criteria were peer-reviewed, full-text studies in English reporting empirical implementation of LLMs in clinical settings. Study quality and risk of bias were assessed using the PROBAST tool. Results: Four studies published between 2024 and 2025 met inclusion criteria. All used generative pre-trained transformers (GPTs). Reported applications included outpatient communication, mental health support, inbox message drafting, and clinical data extraction. LLM deployment was associated with improvements in operational efficiency, user satisfaction, and reduced workload. However, challenges included performance variability across data types, limitations in generalizability, regulatory delays, and lack of post-deployment monitoring. Conclusions: Early evidence suggests that LLMs can enhance clinical workflows, but real-world adoption remains constrained by systemic, technical, and regulatory barriers. To support safe and scalable use, future efforts should prioritize standardized evaluation metrics, multi-site validation, human oversight, and implementation frameworks tailored to clinical settings. Systematic Review Registration: https://www.crd.york.ac.uk/PROSPERO/recorddashboard, PROSPERO CRD420251030069." +W7135053800,,10.1007/s42600-026-00465-3,2026,Large language models in portuguese for healthcare: a systematic review,Andre Massahiro Shimaoka,7,Research on Biomedical Engineering,1,"Abstract Purpose This study addresses Large Language Models (LLMs) pre-trained in Portuguese for healthcare applications, focusing on contextual embeddings. Research on LLMs for natural language processing (NLP) tasks in Portuguese is limited, especially within healthcare. Much of the existing research has focused on high-resource languages such as English. However, LLMs demonstrate potential in clinical decision support, diagnosis assistance, patient care, and other healthcare applications. In view of this, the present work assesses the current state of LLMs in Portuguese for healthcare. Methods Our Systematic Literature Review (SLR) followed standard protocols: search, screening based on inclusion/exclusion criteria, quality assessment, data extraction, and analysis. Results We identified 32 models, mostly based on BERTimbau, mBERT, and BioBERTpt. Adaptation strategies such as fine-tuning, domain-adaptive pre-training, training from scratch, and zero-shot learning have been the most prevalent. Several datasets have been used, including clinical records, social media, and scientific repositories. LLMs in Portuguese are being applied in mental health, general medicine, COVID-19, oncology, and other related areas, accomplishing classification tasks, followed by named entity recognition (NER), topic modeling, question answering, text generation, summarization, de-identification, and conversational agents. Conclusion Our study identified key gaps and opportunities: (1) unexplored recent LLMs such as T5, Qwen, DeepSeek, BART, among others; (2) insufficient fine-tuning details, hindering reproducibility; (3) limited coverage of healthcare fields; (4) clinical and hospital data widely used but not shared; (5) social media data requiring caution due to potential inconsistencies; and (6) overlooked data privacy." +W4390700469,,10.1101/2024.01.05.24300884,2024,Large language models in simplifying radiological reports: systematic review,Yaara Artsi,6,medRxiv,9,"Abstract Objectives Simplifying medical information to make it understandable for patients, specifically in the case of radiology reports, is challenging. It requires time and effort from medical personnel. This systematic review focuses on the application of large language models (LLMs) in generating simplified radiological imaging reports, as well as answering patient inquiries regarding radiological procedures. Materials and Methods The authors searched for studies published up to January 2024. Search terms focused on LLMs generated simplified radiological reports and answers to patient inquiries regarding radiological procedures. MEDLINE was used as a search database. Results Overall, eight studies published between May 2023 and November 2023 were included. All studies showed that LLMs can produce simplified medical information for patients. Four studies (50%) used GPT-3.5, Two studies (25%) conducted a comparative analysis between GPT-3.5 and GPT-4. One study (12.5%) examined Microsoft Bing. One study (12.5%) utilized GPT-4. Four studies (50%) used LLMs to simplify radiological reports. Four studies (50%) used LLMs to answer patient questions regarding radiological procedures. Only two studies (25%) used patients to evaluate the LLMs output. One study (12.5%) compared their initial prompt with optimized prompt. Five studies (62.5%) showed missing, inaccurate and potentially harmful AI outputs. Conclusion LLMs can be used to simplify medical imaging reports and procedures, for improved patient comprehension. However, their limitations cannot be ignored. Further study in this field is essential and more conclusive evidence is needed." +W4390897455,10.48550/arxiv.2401.06320,10.48550/arxiv.2401.06320,2024,Zero-shot Generative Large Language Models for Systematic Review Screening Automation,Shuai Wang,6,arXiv (Cornell University),5,"Systematic reviews are crucial for evidence-based medicine as they comprehensively analyse published research findings on specific questions. Conducting such reviews is often resource- and time-intensive, especially in the screening phase, where abstracts of publications are assessed for inclusion in a review. This study investigates the effectiveness of using zero-shot large language models~(LLMs) for automatic screening. We evaluate the effectiveness of eight different LLMs and investigate a calibration technique that uses a predefined recall threshold to determine whether a publication should be included in a systematic review. Our comprehensive evaluation using five standard test collections shows that instruction fine-tuning plays an important role in screening, that calibration renders LLMs practical for achieving a targeted recall, and that combining both with an ensemble of zero-shot models saves significant screening time compared to state-of-the-art approaches." +W7132993116,,,2025,Prompting Strategies for Large Language Models in Requirements Elicitation : A Systematic Literature Review,Naveed Ali,1,Tampere University Institutional Repository (Tampere University),0,"Recent advancements in Large Language Models (LLMs) such as GPT-4, DeepSeek, Gemini, and LLaMA have introduced new opportunities for supporting requirements elicitation, one of the most knowledge-intensive and error-prone activities in Requirements Engineering (RE). The effectiveness of LLM-assisted elicitation depends largely on how prompts are designed, structured, and contextualized. As a wide range of prompting strategies has emerged, including zero-shot, few-shot, chain-of-thought, role-based, and prompt-chaining techniques, there is a growing need to consolidate current evidence and evaluate their strengths, limitations, and practical relevance. This thesis presents a systematic literature review of 58 peer-reviewed studies investigating prompting strategies used with LLMs for requirements elicitation. The review follows established SLR guidelines and PRISMA 2020 procedures. Studies were retrieved from Scopus, IEEE Xplore, and the ACM Digital Library using a unified search strategy and underwent a structured screening, quality assessment, and data extraction process. The findings categorize prompting strategies across multiple dimensions, including their structural design, reasoning mechanisms, and integration with supporting tools such as retrieval-augmented generation pipelines, conversational agents, and domain-specific knowledge models. The review also synthesizes how prompting is applied across key elicitation contexts. Furthermore, it examines evaluation approaches, highlighting trends in quantitative metrics, expert judgment, and mixed-method assessment. The study contributes a consolidated view of prompting practices in LLM-based elicitation, identifies recurring benefits and challenges, and outlines emerging best practices. By synthesizing methodological patterns and gaps, this review offers guidance for researchers and practitioners seeking to design effective prompting strategies and develop reliable, human-centered elicitation workflows supported by generative AI." +W4407395599,,10.1097/hm9.0000000000000143,2025,Large language models in traditional Chinese medicine: a systematic review,Zhe Chen,8,Acupuncture and Herbal Medicine,16,"Objective: Generative artificial intelligence (AI) technology, represented by large language models (LLMs), has gradually been developed for traditional Chinese medicine (TCM); however, challenges remain in effectively enhancing AI applications for TCM. Therefore, this study is the first systematic review to analyze LLMs in TCM retrospectively, focusing on and summarizing the evidence of their performance in generative tasks. Methods: We extensively searched electronic databases for articles published until June 2024 to identify publicly available studies on LLMs in TCM. Two investigators independently selected and extracted the related information and evaluation metrics. Based on the available data, this study used descriptive analysis for a comprehensive systematic review of LLM technology related to TCM. Results: Ten studies published between 2023 and 2024 met our eligibility criteria and were included in this review, including 40% LLMs in the TCM vertical domain, 40% containing TCM data, and 20% honoring the TCM contribution, with a foundational model parameter range from 1.8 to 33 billion. All included studies used manual or automatic evaluation metrics to evaluate model performance and fully discussed the challenges and contributions through an overview of LLMs in TCM. Conclusions: LLMs have achieved significant advantages in TCM applications and can effectively address intelligent TCM tasks. Further in-depth development of LLMs is needed in various vertical TCM fields, including clinical and fundamental research. Focusing on the functional segmentation development direction of generative AI technologies in TCM application scenarios to meet the practical needs-oriented demands of TCM digitalization is essential." +W7156901103,,10.1002/widm.70089,2026,Large Language Models for Explainable Medical Text Summarization: A Systematic Literature Review,Aleka Melese Ayalew,4,Wiley Interdisciplinary Reviews Data Mining and Knowledge Discovery,3,"ABSTRACT Recently, large language models have gained momentum in several areas, including medical text summarization and clinical decision support in healthcare. In this systematic review, we focus on how LLMs are used for summarization and explainability, and on how they improve the precision of clinical decisions. Initially, by screening eight distinct databases, we identified 1601 studies using the PRISMA protocol. Following strict qualifying criteria, 61 studies were selected for the final review. Our results show that decoder‐only architectures dominate the field (60.7% of studies), with zero‐shot prompting emerging as the predominant approach (55.7%), while 24.6% of studies employed fine‐tuning. General clinical decision support (26.2%) and diagnostic assistance (19.7%) constitute the two most widely used clinical applications across multiple clinical specialties. Identified in 26.2% of studies, the intersection of summarization and explainability becomes a key focus area, with fine‐tuned models obtaining 38.6% higher average ROUGE‐1 scores compared to zero‐shot approaches, and studies including robust explainability features report 27.3% higher clinician acceptance rates. Despite great potential, major obstacles still exist, including model hallucinations (63.9% of studies), minimal workflow integration (only 14.8% of studies), and inadequate attention to regulatory paths (8.2%). Although LLMs offer significant potential for transforming clinical decision support via improved information handling and clear reasoning, realizing these goals requires overcoming essential challenges in clinical validation, system integration, and ethical oversight. This review aims to provide an extensive framework for understanding the current capabilities, limitations, and prospects of LLM‐based explainable summarization in clinical settings, guiding future research and practical deployment. This article is categorized under: Technologies > Artificial Intelligence Fundamental Concepts of Data and Knowledge > Explainable AI Application Areas > Health Care" +W4410090731,,10.2196/preprints.76734,2025,Large Language Models in Cardiology: A Systematic Review (Preprint),Moran Gendler,8,,0,"UNSTRUCTURED Purpose: This review analyzes the application of large language models (LLMs) in the field of cardiology, with a focus on evaluating their performances across various clinical tasks. Methods: We conducted a systematic literature search on PubMed for studies published up to April 14, 2024. Our search strategy involved a combination of multiple keywords pertaining to LLMs and various aspects of cardiology. The risk of bias was evaluated using the QUADAS-2 tool. Results: Fifteen studies met the inclusion criteria, categorized into four domains: chronic and progressive cardiac conditions, acute cardiac events, cardiology education, and cardiac monitoring. Six studies addressing chronic conditions demonstrated variability in the accuracy and depth of LLM-generated responses. In acute cardiac scenarios, three articles showed that LLMs provided medical advice with mixed effectiveness, particularly in delivering CPR instructions. Two studies in educational cardiology revealed high accuracy in answering assessment questions and interpreting clinical cases. Finally, four articles on cardiac diagnostics showed that multimodal LLMs displayed a range of capabilities in ECGs interpretation, with some models performing at or exceeding the level of human specialists. Conclusion: LLMs demonstrate considerable potential in the field of cardiology, particularly in educational applications and routine diagnostics. However, their performance remains inconsistent across various clinical scenarios, particularly in acute care settings where precision is critical. Enhancing LLMs' accuracy in interpreting real-world complex medical data and emergency response guidance is imperative before integration into clinical practice." +W4409970142,,10.7717/peerj-cs.2822,2025,Validation of automated paper screening for esophagectomy systematic review using large language models,Rashi Ramchandani,10,PeerJ Computer Science,5,"Background: Large language models (LLMs) offer a potential solution to the labor-intensive nature of systematic reviews. This study evaluated the ability of the GPT model to identify articles that discuss perioperative risk factors for esophagectomy complications. To test the performance of the model, we tested GPT-4 on narrower inclusion criterion and by assessing its ability to discriminate relevant articles that solely identified preoperative risk factors for esophagectomy. Methods: = 1,967) discussing risk factors to esophagectomy complications. The articles underwent title and abstract screening by three independent human reviewers and GPT-4. The Python script used for the analysis made Application Programming Interface (API) calls to GPT-4 with screening criteria in natural language. GPT-4's inclusion and exclusion decision were compared to those decided human reviewers. Results: The agreement between the GPT model and human decision was 85.58% for perioperative factors and 78.75% for preoperative factors. The AUC value was 0.87 and 0.75 for the perioperative and preoperative risk factors query, respectively. In the evaluation of perioperative risk factors, the GPT model demonstrated a high recall for included studies at 89%, a positive predictive value of 74%, and a negative predictive value of 84%, with a low false positive rate of 6% and a macro-F1 score of 0.81. For preoperative risk factors, the model showed a recall of 67% for included studies, a positive predictive value of 65%, and a negative predictive value of 85%, with a false positive rate of 15% and a macro-F1 score of 0.66. The interobserver reliability was substantial, with a kappa score of 0.69 for perioperative factors and 0.61 for preoperative factors. Despite lower accuracy under more stringent criteria, the GPT model proved valuable in streamlining the systematic review workflow. Preliminary evaluation of inclusion and exclusion justification provided by the GPT model were reported to have been useful by study screeners, especially in resolving discrepancies during title and abstract screening. Conclusion: This study demonstrates promising use of LLMs to streamline the workflow of systematic reviews. The integration of LLMs in systematic reviews could lead to significant time and cost savings, however caution must be taken for reviews involving stringent a narrower and exclusion criterion. Future research is needed and should explore integrating LLMs in other steps of the systematic review, such as full text screening or data extraction, and compare different LLMs for their effectiveness in various types of systematic reviews." +W7172313156,,10.21203/rs.3.rs-10387804/v1,2026,Mathematical Scientific Discovery Using Large Language Models: A Systematic Literature Review,Vinita Gangaram Jansari,6,Research Square,0, +W4399327517,,10.1101/2024.06.03.24308405,2024,Evaluating the Efficacy of Large Language Models for Systematic Review and Meta-Analysis Screening,Ronald Luo,4,medRxiv,8,"ABSTRACT Background Systematic reviews and meta-analyses are essential for informed research and policymaking, yet they are typically resource-intensive and time-consuming. Recent advances in artificial intelligence and machine learning offer promising opportunities to streamline these processes. Objective To enhance the efficiency of systematic reviews, we explored the automation of various stages using GPT-3.5 Turbo. We assessed the model’s efficacy and performance by comparing it against three expert-conducted reviews across a comprehensive dataset of 24,534 studies. Methods The model’s performance was evaluated through a comparison with three expert reviews, utilizing a pseudo-K-folds permutation and a one-tailed ANOVA with an alpha level of 0.05 to ensure statistical validity. Key performance metrics such as accuracy, sensitivity, specificity, predictive values, F1-score, and the Matthews correlation coefficient were analyzed using two sets of prompts. Results Our approach significantly streamlined the systematic review process, which typically takes a year, reducing it to a few hours without sacrificing quality. In the initial screening phase, accuracy, specificity, and negative predictive values ranged between 80% and 95%. Sensitivity improved markedly during the second screening phase, demonstrating the model’s robustness when provided with more extensive data. Conclusion While ongoing refinements are needed, this tool represents a significant advancement in research methodologies, potentially making systematic reviews more accessible to a wider range of researchers. Impact Statement Our manuscript presents a novel review screening protocol built using open-source frameworks, which significantly enhances the systematic review process in terms of efficiency and cost-effectiveness. Leveraging the capabilities of GPT and embedding models, our protocol demonstrates the potential to transform a traditionally time-consuming and expensive task into an accelerated and economical operation, all while maintaining high standards of accuracy and reliability. Key Points GPT screening can streamline systematic reviews from a year-long, expensive process to just hours at minimal cost. Validated across different topics, the protocol exhibits high reliability and consistency in study inclusion. The AI-driven process reduces human bias, with prompt optimization considerably improving sensitivity." +W7152009027,,10.3390/jcm15082830,2026,Efficacy of Large Language Models for Screening of Systematic Reviews on Periprosthetic Joint Infection,Woojin Shin,6,Journal of Clinical Medicine,0,"Background: Periprosthetic joint infection (PJI) remains a devastating complication following arthroplasty. Systematic reviews of PJI provide essential evidence to inform clinical practice; however, the screening process remains labor-intensive. Recent advancements in large language models (LLMs) offer potential for automating literature screening, though evaluation of current generation models is needed. Methods: This validation study evaluated GPT-5, GPT-5 Pro, and Gemini 2.5 Pro in replicating the title/abstract and full-text screening stages of a published systematic review on intraosseous versus intravenous antibiotic prophylaxis in total joint arthroplasty. Title/abstract screening was performed on 165 articles, followed by a full-text eligibility assessment of 26 articles. Accuracy, sensitivity, specificity, and Cohen’s kappa (κ) were calculated against human screening decisions as the gold standard. Results: In title/abstract screening, GPT-5 Pro achieved the highest accuracy (92.1–92.7%) and specificity (98.6–99.3%), while GPT-5 demonstrated the highest sensitivity (84.6–96.1%). In full-text screening, Gemini 2.5 Pro showed the most consistent performance across repeated evaluations (κ = 0.839 in both trials), whereas GPT-5 Pro exhibited marked intra-model variability (κ = 0.399 to 0.920). Conclusions: Current-generation LLMs achieve near-human accuracy in systematic review screening for PJI research, though substantial intra-model variability underscores the continued need for human oversight in systematic review workflows." +W4405183884,,10.3390/jcm13237480,2024,Exploring the Role of Large Language Models in Melanoma: A Systematic Review,Mor Zarfati,7,Journal of Clinical Medicine,9,"Objective: This systematic review evaluates the current applications, advantages, and challenges of large language models (LLMs) in melanoma care. Methods: A systematic search was conducted in PubMed and Scopus databases for studies published up to 23 July 2024, focusing on the application of LLMs in melanoma. The review adhered to PRISMA guidelines, and the risk of bias was assessed using the modified QUADAS-2 tool. Results: Nine studies were included, categorized into subgroups: patient education, diagnosis, and clinical management. In patient education, LLMs demonstrated high accuracy, though readability often exceeded recommended levels. For diagnosis, multimodal LLMs like GPT-4V showed capabilities in distinguishing melanoma from benign lesions, but accuracy varied, influenced by factors such as image quality and integration of clinical context. Regarding management advice, ChatGPT provided more reliable recommendations compared to other LLMs, but all models lacked depth for individualized decision-making. Conclusions: LLMs, particularly multimodal models, show potential in improving melanoma care. However, current applications require further refinement and validation. Future studies should explore fine-tuning these models on large, diverse dermatological databases and incorporate expert knowledge to address limitations such as generalizability across different populations and skin types." +W4394855484,,10.3390/healthcare12080825,2024,AI and Ethics: A Systematic Review of the Ethical Considerations of Large Language Model Use in Surgery Research,Sophia M. Pressman,6,Healthcare,111,"INTRODUCTION: As large language models receive greater attention in medical research, the investigation of ethical considerations is warranted. This review aims to explore surgery literature to identify ethical concerns surrounding these artificial intelligence models and evaluate how autonomy, beneficence, nonmaleficence, and justice are represented within these ethical discussions to provide insights in order to guide further research and practice. METHODS: A systematic review was conducted in accordance with the Preferred Reporting Items for Systematic Reviews and Meta-Analyses guidelines. Five electronic databases were searched in October 2023. Eligible studies included surgery-related articles that focused on large language models and contained adequate ethical discussion. Study details, including specialty and ethical concerns, were collected. RESULTS: The literature search yielded 1179 articles, with 53 meeting the inclusion criteria. Plastic surgery, orthopedic surgery, and neurosurgery were the most represented surgical specialties. Autonomy was the most explicitly cited ethical principle. The most frequently discussed ethical concern was accuracy (n = 45, 84.9%), followed by bias, patient confidentiality, and responsibility. CONCLUSION: The ethical implications of using large language models in surgery are complex and evolving. The integration of these models into surgery necessitates continuous ethical discourse to ensure responsible and ethical use, balancing technological advancement with human dignity and safety." +W7220675612,,10.21203/rs.3.rs-11150018/v1,2026,An Auditable Large Language Model Pipeline for Data Extraction in Systematic Literature Reviews,Philipp André Macháček,5,Research Square,0, +W4412490346,,10.1038/s41746-025-01824-7,2025,Large language model integrations in cancer decision-making: a systematic review and meta-analysis,Yuexing Hao,7,npj Digital Medicine,66,"Large Language Models (LLMs) are increasingly used to support cancer patients and clinicians in decision-making. This systematic review investigates how LLMs are integrated into oncology and evaluated by researchers. We conducted a comprehensive search across PubMed, Web of Science, Scopus, and the ACM Digital Library through May 2024, identifying 56 studies covering 15 cancer types. The meta-analysis results suggested that LLMs were commonly used to summarize, translate, and communicate clinical information, but performance varied: the average overall accuracy was 76.2%, with average diagnostic accuracy lower at 67.4%, revealing gaps in the clinical readiness of this technology. Most evaluations relied heavily on quantitative datasets and automated methods without human graders, emphasizing ""accuracy"" and ""appropriateness"" while rarely addressing ""safety"", ""harm"", or ""clarity"". Current limitations for LLMs in cancer decision-making, such as limited domain knowledge and dependence on human oversight, demonstrate the need for open datasets and standardized evaluations to improve reliability." +W4391932545,,10.2196/preprints.57400,2024,Large Language Models for Mental Health Applications: Systematic Review (Preprint),Zhijun Guo,6,,24,"BACKGROUND Large language models (LLMs) are advanced artificial neural networks trained on extensive datasets to accurately understand and generate natural language. While they have received much attention and demonstrated potential in digital health, their application in mental health, particularly in clinical settings, has generated considerable debate. OBJECTIVE This systematic review aims to critically assess the use of LLMs in mental health, specifically focusing on their applicability and efficacy in early screening, digital interventions, and clinical settings. By systematically collating and assessing the evidence from current studies, our work analyzes models, methodologies, data sources, and outcomes, thereby highlighting the potential of LLMs in mental health, the challenges they present, and the prospects for their clinical use. METHODS Adhering to the PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) guidelines, this review searched 5 open-access databases: MEDLINE (accessed by PubMed), IEEE Xplore, Scopus, JMIR, and ACM Digital Library. Keywords used were (mental health OR mental illness OR mental disorder OR psychiatry) AND (large language models). This study included articles published between January 1, 2017, and April 30, 2024, and excluded articles published in languages other than English. RESULTS In total, 40 articles were evaluated, including 15 (38%) articles on mental health conditions and suicidal ideation detection through text analysis, 7 (18%) on the use of LLMs as mental health conversational agents, and 18 (45%) on other applications and evaluations of LLMs in mental health. LLMs show good effectiveness in detecting mental health issues and providing accessible, destigmatized eHealth services. However, assessments also indicate that the current risks associated with clinical use might surpass their benefits. These risks include inconsistencies in generated text; the production of hallucinations; and the absence of a comprehensive, benchmarked ethical framework. CONCLUSIONS This systematic review examines the clinical applications of LLMs in mental health, highlighting their potential and inherent risks. The study identifies several issues: the lack of multilingual datasets annotated by experts, concerns regarding the accuracy and reliability of generated content, challenges in interpretability due to the “black box” nature of LLMs, and ongoing ethical dilemmas. These ethical concerns include the absence of a clear, benchmarked ethical framework; data privacy issues; and the potential for overreliance on LLMs by both physicians and patients, which could compromise traditional medical practices. As a result, LLMs should not be considered substitutes for professional mental health services. However, the rapid development of LLMs underscores their potential as valuable clinical aids, emphasizing the need for continued research and development in this area. CLINICALTRIAL PROSPERO CRD42024508617; https://www.crd.york.ac.uk/prospero/display_record.php?RecordID=508617" +W7117820692,,10.3390/app16010368,2025,Retrieval-Augmented Generation (RAG) and Large Language Models (LLMs) for Enterprise Knowledge Management and Document Automation: A Systematic Literature Review,Ehlullah Karakurt,2,Applied Sciences,25,"The integration of Retrieval-Augmented Generation (RAG) with Large Language Models (LLMs) is rapidly transforming enterprise knowledge management, yet a comprehensive understanding of their deployment in real-world workflows remains limited. This study presents a systematic literature review (SLR) analyzing 63 high-quality primary studies selected after rigorous screening to evaluate how these technologies address practical enterprise challenges. We formulated nine research questions targeting platforms, datasets, algorithms, and validation metrics to map the current landscape. Our findings reveal that enterprise adoption is largely in the experimental phase: 63.6% of implementations utilize GPT based models, and 80.5% rely on standard retrieval frameworks such as FAISS or Elasticsearch. Critically, this review identifies a significant ‘lab-to-market’ gap; while retrieval and classification sub-tasks frequently employ academic validation methods like k-fold cross-validation (93.6%), generative evaluation predominantly relies on static hold-out sets due to computational constraints. Furthermore, fewer than 15% of studies address real-time integration challenges required for production scale deployment. By systematically mapping these disparities, this study offers a data-driven perspective and a strategic roadmap for bridging the gap between academic prototypes and robust enterprise applications." +W4388331108,,10.1101/2023.11.04.23298081,2023,Applications of Large Language Models (LLMs) in Breast Cancer Care,Vera Sorin,6,medRxiv,11,"Abstract Purpose Recently introduced Large Language Models (LLMs) such as ChatGPT have already shown promising results in natural language processing in healthcare. The aim of this study is to systematically review the literature on the applications of LLMs in breast cancer diagnosis and care. Methods A literature search was conducted using MEDLINE, focusing on studies published up to October 22nd, 2023, using the following terms: “large language models”, “LLM”, “GPT”, “ChatGPT”, “OpenAI”, and “breast”. Results Five studies met our inclusion criteria. All studies were published in 2023, focusing on ChatGPT-3.5 or GPT-4 by OpenAI. Applications included information extraction from clinical notes, question-answering based on guidelines, and patients’ management recommendations. The rate of correct answers varied from 64-98%, with the highest accuracy (88-98%) observed in information extraction and question-answering tasks. Notably, most studies utilized real patient data rather than data sourced from the internet. Limitations included inconsistent accuracy, prompt sensitivity, and overlooked clinical details, highlighting areas for cautious LLM integration into clinical practice. Conclusion LLMs demonstrate promise in text analysis tasks related to breast cancer care, including information extraction and guideline-based question-answering. However, variations in accuracy and the occurrence of erroneous outputs necessitate validation and oversight. Future works should focus on improving reliability of LLMs within clinical workflow." +W4414365239,,10.1093/ckj/sfaf243,2025,Clinical applications and limitations of large language models in nephrology: a systematic review,Zoe Unger,6,Clinical Kidney Journal,12,"ABSTRACT Background Large language models (LLMs) have emerged as potential tools in healthcare. This systematic review evaluates the applications of text-generative conversational LLMs in nephrology, with particular attention to their reported advantages and limitations. Methods A systematic search was performed in PubMed, Web of Science, Embase and the Cochrane Library in accordance with the Preferred Reporting Items for Systematic Reviews and Meta-Analyses guidelines. Eligible studies assessed LLM applications in nephrology. PROSPERO registration number CRD42024550169. Results Of 1070 records screened, 23 studies met inclusion criteria, addressing four clinical applications in nephrology. In patient education (n = 13), GPT-4 improved the readability of kidney donation information from a 10th to a 4th grade level (9.6 ± 1.9 to 4.30 ± 1.71) and Gemini provided the most accurate answers to chronic kidney disease questions (Global Quality Score 3.46 ± 0.55). Regarding workflow optimization (n = 7), GPT-4 achieved high accuracy (90–94%) in managing continuous renal replacement therapy alarms and improved diagnosis of diabetes insipidus using chain-of-thought and retrieval-augmented prompting. In renal dietary guidance (n = 2), Bard AI led in classifying phosphorus and oxalate content of foods (100% and 84%), while GPT-4 and Bing Chat were most accurate for potassium classification (81%). For laboratory data interpretation (n = 1), Copilot significantly outperformed ChatGPT and Gemini in simulated nephrology datasets (median scores 5/5 compared with 4/5 and 4/5; P < .01). TRIPOD-LLM assessment revealed frequent omissions in data handling, prompting strategies and transparency. Conclusions While LLMs may enhance various aspects of nephrology practice, their widespread adoption remains premature. Input-quality dependence and limited external validation restrict generalizability. Further research is needed to confirm their real-world feasibility and ensure safe clinical integration." +W7156817509,,10.5281/zenodo.19856513,2026,Prompt injection attacks and defenses in large language models: A systematic literature review,Carmine Ambrosino,1,Zenodo (CERN European Organization for Nuclear Research),0,"Prompt injection has rapidly emerged as a critical security threat in the deployment of large language models (LLMs), enabling adversaries to subvert intended behaviors and bypass safety mechanisms. Despite the increased attention that this threat has received, no previous studies have systematically analyzed the field. This paper presents the first systematic literature review (SLR) on prompt injection, with the objective of facilitating a comprehensive, evidence-based understanding of the existing attacks and defenses in LLM among researchers and practitioners.We extensively searched databases like ACM Digital Library, ScienceDirect and Web of Science, initially screening 207 studies and ultimately focusing on 56 relevant papers, based on rigorous inclusion, exclusion, and quality criteria. The analysis is structured around three core research questions: (i) the taxonomical classification of prompt injection attacks, (ii) the identification of recent and innovative attack techniques, and (iii) the characterization of proposed defense mechanisms. The findings reveal a rapidly evolving and multi-layered threat landscape, encompassing obfuscation strategies, automated and multi-modal attacks, and psychologically manipulative prompts. In response, the literature proposes a range of defenses, including input-level sanitization, model-level filtering, prompt engineering, classification-based approaches, and architectural safeguards. Future research should focus on establishing robust standardization in both theory and experimentation, addressing the heterogeneity in attack classification and defense evaluation, while promoting empirical and quantitative approaches to assess effectiveness and considering user privacy and ethical implications." +W7130360482,,10.21203/rs.3.rs-8902070/v1,2026,Large Language Models in Clinical Neurology: A Systematic Review,Alon Gorenshtein,6,Research Square,2, +W4387144848,,10.2196/48996,2023,Automated Paper Screening for Clinical Reviews Using Large Language Models: Data Analysis Study,Eddie Guo,6,Journal of Medical Internet Research,205,"BACKGROUND: The systematic review of clinical research papers is a labor-intensive and time-consuming process that often involves the screening of thousands of titles and abstracts. The accuracy and efficiency of this process are critical for the quality of the review and subsequent health care decisions. Traditional methods rely heavily on human reviewers, often requiring a significant investment of time and resources. OBJECTIVE: This study aims to assess the performance of the OpenAI generative pretrained transformer (GPT) and GPT-4 application programming interfaces (APIs) in accurately and efficiently identifying relevant titles and abstracts from real-world clinical review data sets and comparing their performance against ground truth labeling by 2 independent human reviewers. METHODS: We introduce a novel workflow using the Chat GPT and GPT-4 APIs for screening titles and abstracts in clinical reviews. A Python script was created to make calls to the API with the screening criteria in natural language and a corpus of title and abstract data sets filtered by a minimum of 2 human reviewers. We compared the performance of our model against human-reviewed papers across 6 review papers, screening over 24,000 titles and abstracts. RESULTS: -score of 0.60, a sensitivity of excluded papers of 0.91, and a sensitivity of included papers of 0.76. The interrater variability between 2 independent human screeners was κ=0.46, and the prevalence and bias-adjusted κ between our proposed methods and the consensus-based human decisions was κ=0.96. On a randomly selected subset of papers, the GPT models demonstrated the ability to provide reasoning for their decisions and corrected their initial decisions upon being asked to explain their reasoning for incorrect classifications. CONCLUSIONS: Large language models have the potential to streamline the clinical review process, save valuable time and effort for researchers, and contribute to the overall quality of clinical reviews. By prioritizing the workflow and acting as an aid rather than a replacement for researchers and reviewers, models such as GPT-4 can enhance efficiency and lead to more accurate and reliable conclusions in medical research." +W7133721140,,10.3390/info17030258,2026,"Descriptive Heterogeneity of the Hornblower Sign Across Scientific Literature, Search Engines, and Large Language Models",Peter Melcher,4,Information,0,"Background/Objectives: Digitalization of medical knowledge has improved access to information but also increased the spread of imprecise content. Repeated exposure to incorrect descriptions may lead to their normalization over time. This is particularly evident for the Hornblower sign, which is frequently conflated with the Patte test in the literature, online sources, and large language model outputs. This study systematically evaluates these descriptions and quantifies related inaccuracies. Methods: A three-step approach was applied to answer the question “What is the Hornblower sign?”. First, primary publications referring to the original description by Arthuis were analyzed. Second, the first 35 Google search results were systematically reviewed. Third, responses from five widely used LLMs (ChatGPT 5.1, Grok 4.1, Gemini 3 Pro, Perplexity, and DeepSeek-R1) were evaluated. All descriptions were assessed using a standardized 4-point scoring system (0–3 points) capturing content accuracy and correct differentiation between the Hornblower sign and the Patte test. Results: Fourteen original publications were included, yielding a mean score of 2.07. Correct descriptions were found in 50%, while 43% described only the Patte test. Among 34 evaluable Google search results, the mean score was 1.17, with 77% scoring ≤ 1 point. The five LLMs achieved a mean score of 1.8, demonstrating substantial variability and overall incomplete conceptual accuracy. Conclusions: Descriptions of the Hornblower sign show substantial heterogeneity and frequent inaccuracies across the scientific literature, online sources, and LLM outputs. Conflation with the Patte test undermines diagnostic reliability and limits study comparability. Critical source appraisal and adherence to original test descriptions are essential to maintain clinical and scientific rigor." +W7166039827,,10.1016/j.jval.2026.03.1578,2026,MSR79 AUTOMATION OF SYSTEMATIC REVIEWS WITH LARGE LANGUAGE MODELS,Christian Cao,1,Value in Health,0, +W7167699919,,10.20944/preprints202607.0296.v1,2026,"A Systematic Review of Large Language Models, 2017 to 2026",Prince Samuel Kyeremanteng,3,Preprints.org,0,"Large language models (LLMs) have moved from research artifacts to infrastructure for search, programming, analytics, education, health, public services, and enterprise decision support. The literature has grown quickly, but the evidence is uneven. Some papers report clear gains from scale, instruction tuning, retrieval, and tool use. Others show brittle reasoning, benchmark leakage, privacy exposure, harmful outputs, and weak documentation. This systematic review maps the technical and governance evidence on LLMs from 2017 to 2026 using a PRISMA 2020 guided process. Searches were conducted in OpenAlex and Crossref on June 5, 2026, with citation chasing for seminal model, benchmark, alignment, retrieval, agent, and risk papers. The final synthesis includes 85 validated studies, each checked through DOI, arXiv, ACL Anthology, ACM, OpenReview, publisher, or official technical-report metadata. Results are organized around ten themes: architecture and scaling, data and documentation, alignment and adaptation, prompting and reasoning, retrieval and grounding, tools and agents, evaluation, risk and governance, domain-specialized models, and multimodal extensions. The review finds that LLM progress is not explained by parameter count alone. Capability depends on data composition, compute allocation, post-training, interfaces, retrieval design, evaluation protocol, and deployment controls. For industry, the most defensible path is not blind adoption or blanket rejection. It is task specific evaluation, provenance-aware retrieval, staged release, documented failure handling, privacy testing, human accountability, and continuous monitoring. The paper closes with a research agenda for evidence that is more reproducible, deployment-aware, and useful outside leaderboards." +W4414834072,10.48550/arxiv.2502.15702,10.48550/arxiv.2502.15702,2025,Large language models streamline automated systematic review: A preliminary study,Xi Chen,2,arXiv (Cornell University),2,"Large Language Models (LLMs) have shown promise in natural language processing tasks, with the potential to automate systematic reviews. This study evaluates the performance of three state-of-the-art LLMs in conducting systematic review tasks. We assessed GPT-4, Claude-3, and Mistral 8x7B across four systematic review tasks: study design formulation, search strategy development, literature screening, and data extraction. Sourced from a previously published systematic review, we provided reference standard including standard PICO (Population, Intervention, Comparison, Outcome) design, standard eligibility criteria, and data from 20 reference literature. Three investigators evaluated the quality of study design and eligibility criteria using 5-point Liker Scale in terms of accuracy, integrity, relevance, consistency and overall performance. For other tasks, the output is defined as accurate if it is the same as the reference standard. Search strategy performance was evaluated through accuracy and retrieval efficacy. Screening accuracy was assessed for both abstracts screening and full texts screening. Data extraction accuracy was evaluated across 1,120 data points comprising 3,360 individual fields. Claude-3 demonstrated superior overall performance in PICO design. In search strategy formulation, GPT-4 and Claude-3 achieved comparable accuracy, outperforming Mistral. For abstract screening, GPT-4 achieved the highest accuracy, followed by Mistral and Claude-3. In data extraction, GPT-4 significantly outperformed other models. LLMs demonstrate potential for automating systematic review tasks, with GPT-4 showing superior performance in search strategy formulation, literature screening and data extraction. These capabilities make them promising assistive tools for researchers and warrant further development and validation in this field." +W4416589125,,10.1016/j.ijmedinf.2025.106205,2025,AI-assisted literature screening: A hybrid approach using large language models and retrieval-augmented generation,Yiming Li,10,International Journal of Medical Informatics,8, +W7117516269,,10.1109/cinti67731.2025.11311831,2025,Large Language Models for Title/Abstract Screening in Systematic Literature Reviews: A Case Study in Precision Livestock Farming,Márk Zrubka,4,IEEE International Symposium on Computational Intelligence and Informatics (CINTI),2,"Large language models (LLMs) are gaining popularity in systematic literature review automation, and they are even acknowledged as second reviewers along humans by the Cochrane Handbook. In this study, we tested the performance of two LLMs in the title/abstract screening process in a systematic review on precision livestock farming costbenefit analyses. We assessed the screening accuracy, sensitivity and specificity of the GPT-4o Mini and Claude 3.5 Sonnet models, as well as a human rater. We also compared the time and costs of completing the title/abstract screening by two human raters versus one human assisted by LLMs. LLMassisted screening required less time and money than two human raters, but it was also less accurate, resulting in extra work to rescreen discrepancies. Nevertheless, combining LLMs with the human ratings improved the quality of the screening output. Our results suggest that the tested LLMs are not as reliable as expert humans, however, they can assist in the title/abstract screening process and improve the quality of work with very low extra resources." +W7203580816,,10.5281/zenodo.21967539,2026,Large Language Models as Research Assistants: A Cross-Disciplinary Systematic Literature Review,Indian Institute of Technology Bombay,1,Zenodo (CERN European Organization for Nuclear Research),0,"Large language models have been absorbed into research practice across nearly every discipline in under four years, faster than the evidence base needed to evaluate them could be assembled. This review synthesises 124 sources published between January 2020 and June 2026, identified through structured searches of seven databases, and organises the evidence by research lifecycle stage rather than by model architecture. Randomised evidence shows substantial gains on bounded, verifiable, language-intensive work - a forty per cent reduction in task time in one experiment, twelve per cent more tasks completed at twenty-five per cent greater speed in another - with effects concentrated among initially lower-performing users, alongside sharp degradation on tasks just beyond the model's competence. Reliability failures are structural rather than incidental: fabricated bibliographic references fell from roughly fifty-five per cent in GPT-3.5 to eighteen per cent in GPT-4, an improvement far short of what unverified citation practice requires, while corpus analysis indicates at least 13.5 per cent of 2024 biomedical abstracts show signs of machine assistance. Governance has converged on three norms - no machine authorship, mandatory disclosure, and exclusion from confidential peer review - while leaving verification, provenance and researcher training largely unaddressed. The review proposes a five-level scheme of research autonomy and a verification protocol calibrated to the cost of undetected error, and argues that the field's central deficit is evaluative: benchmarks measure question answering, not whether assisted research is better research." +W7171006779,,,2026,MechaScreener: Large Language Model-Based Automated Screening for Systematic Reviews and Research,Carly Hudson,5,OSF Preprints (OSF Preprints),0,"Systematic Reviews (SRs) are the gold standard for evidence synthesis, but the manual title and abstract screening of thousands of references creates a severe bottleneck. Existing automated tools have historically struggled to achieve the near-perfect recall (sensitivity) required for reliable reviews. We developed MechaScreener as a ""zero-shot"" automated screening tool that utilises a Large Language Model (LLM) to rank article relevance. The tool requires no initial training data or manual pre-screening, as MechaScreener directly applies user-provided question elements (PICO) or inclusion/exclusion criteria to assign an ordinal relevance score (1-5) to each reference. We evaluated the tool in two phases: a development phase using five reference libraries to optimise prompts, and an independent evaluation phase using 10 diverse Cochrane review libraries (comprising both randomised controlled trials and non-RCTs) containing over 58,000 references. In the evaluation dataset, MechaScreener achieved a perfect recall of 1.00 (100\%) in all 10 reviews, ensuring no relevant articles were missed (pooled record-level 95% CI: 0.98-1.00). Concurrently, it achieved an overall macro-average specificity of 0.61 (61%, cluster bootstrap 95% CI: 0.46-0.75; pooled record-level specificity 0.59, 95% CI: 0.59-0.60). Specificity varied: from 0.21 in broad public health topics to 0.91 in precise pharmacological interventions; reflecting the tool's built-in conservatism when evaluating ambiguous abstracts. By safely eliminating over 60\% of irrelevant literature during the initial screening phase without compromising recall, MechaScreener functions as a highly reliable but low-effort ``first-pass'' filter, allowing researchers to substantially reduce manual workloads and reallocate resources toward full-text review and data extraction." +W7170048440,,10.17605/osf.io/u43m2,2026,MechaScreener: Large Language Model-Based Automated Screening for Systematic Reviews and Research,Connor Forbes,5,Open Science Framework,0,"Systematic Reviews (SRs) are the gold standard for evidence synthesis, but the manual title and abstract screening of thousands of references creates a severe bottleneck. Existing automated tools have historically struggled to achieve the near-perfect recall (sensitivity) required for reliable reviews. We developed MechaScreener as a ""zero-shot"" automated screening tool that utilises a Large Language Model (LLM) to rank article relevance. The tool requires no initial training data or manual pre-screening, as MechaScreener directly applies user-provided question elements (PICO) or inclusion/exclusion criteria to assign an ordinal relevance score (1-5) to each reference. We evaluated the tool in two phases: a development phase using five reference libraries to optimise prompts, and an independent evaluation phase using 10 diverse Cochrane review libraries (comprising both randomised controlled trials and non-RCTs) containing over 58,000 references. In the evaluation dataset, MechaScreener achieved a perfect recall of 1.00 (100\%) in all 10 reviews, ensuring no relevant articles were missed (pooled record-level 95% CI: 0.98-1.00). Concurrently, it achieved an overall macro-average specificity of 0.61 (61%, cluster bootstrap 95% CI: 0.46-0.75; pooled record-level specificity 0.59, 95% CI: 0.59-0.60). Specificity varied: from 0.21 in broad public health topics to 0.91 in precise pharmacological interventions; reflecting the tool's built-in conservatism when evaluating ambiguous abstracts. By safely eliminating over 60\% of irrelevant literature during the initial screening phase without compromising recall, MechaScreener functions as a highly reliable but low-effort ``first-pass'' filter, allowing researchers to substantially reduce manual workloads and reallocate resources toward full-text review and data extraction." +W7161880413,,10.3389/or.2026.1757059,2026,Applications of large language models in tumor boards: a systematic review,Benjamin Konzmann,4,Oncology Reviews,4,"Background: Multidisciplinary tumor boards (MDTs) are the gold standard for cancer care but currently face significant pressure from rising case volumes and the increasing complexity of precision medicine. Large Language Models (LLMs) offer potential as Clinical Decision Support Systems (CDSS) to augment these workflows. This systematic review evaluates the current applications, accuracy, and safety of LLMs in MDT decision-making. Methods: A systematic review was conducted following PRISMA 2020 guidelines. We searched PubMed/MEDLINE, Embase, IEEE Xplore, and arXiv for primary research published between January 2023 and November 2025. Inclusion criteria required studies to evaluate LLM performance in tumor board settings against human consensus or established clinical guidelines. Results: Thirty-one studies encompassing 3,845 unique patient cases were included. The analysis revealed a distinct ""complexity gap"" in model performance. In standardized, high-incidence domains such as breast and prostate cancer, advanced models (particularly the GPT-4 family) demonstrated high concordance (up to 94%) with human experts. However, performance degraded significantly in complex, rare, or multimodal scenarios-such as sarcoma or neuro-oncology-where models struggled with ""gray zone"" decision-making and lacked the ability to interpret non-textual data. While LLMs showed utility in administrative tasks and guideline retrieval, they remained prone to hallucinations and lacked the nuance required for holistic patient assessment. Conclusion: Current LLMs exhibit sufficient maturity to function as assistive tools for documentation and decision support in routine oncological cases but are not yet reliable enough for autonomous decision-making. Successful clinical implementation will require ""human-in-the-loop"" safeguards, the development of multimodal architectures, and rigorous prospective validation to ensure patient safety." +W4416534882,,10.1016/j.infsof.2026.108204,2026,LLM4SCREENLIT: Recommendations on assessing the performance of large language models for screening literature in systematic reviews,Lech Madeyski,3,Information and Software Technology,4,"Context: Large language models (LLMs) are increasingly used to screen literature for systematic reviews (SRs), but the standard confusion-matrix metrics used to evaluate them can mislead under the imbalanced, cost-asymmetric conditions of screening. Objective: We develop and justify LLM4SCREENLIT—practical recommendations for researchers conducting LLM-screening evaluations and for editors and reviewers assessing such studies—differentiated by study type (retrospective benchmarking vs. deployment for a specific SR). Method: Using Delgado-Chaves et al. (2025), an 18-LLM benchmark across three biomedical SRs, as a motivating example, we reviewed 28 additional papers and extracted their reported metrics. We propose a Weighted Matthews Correlation Coefficient (WMCC) that integrates MCC’s chance-correction with asymmetric misclassification costs, and validated it on three software-engineering (SE) reanalyses (Felizardo et al. 2024; Syriani et al. 2024; Huotala et al. 2025), the largest covering 9 LLMs × 24 SE secondary studies (34,528 articles). Results: Across the 29 papers, only 10% reported MCC, only 24% reported full confusion matrices, and none of the five papers claiming workload savings priced false-negative cost. In the largest SE reanalysis, MCC and WMCC disagree on the best LLM in 55% of evaluable studies; in the most striking 9695-article SE study, the Accuracy-best LLM loses 63.3% of relevant evidence (Lost Evidence), the MCC-best 43.9%, but the WMCC-best only 5.8%. Sensitivity analysis (median crossover at w ≈ 2 . 7 , all < 7 ) supports w = 10 as a conservative default. Conclusions: SR-screening evaluations should prioritize Lost Evidence and use cost-sensitive WMCC alongside MCC for ranking. Reporting must include the full confusion matrix and treat unclassifiable outputs as positives requiring human review. Designs should be leakage-aware, with non-LLM baselines when the study aims to inform SR practice and labels are available. Editors and reviewers should require these elements as routine. Extension to full-text screening and data extraction is principled but pending empirical validation." +W4410564944,,10.3390/info16050420,2025,PROMPTHEUS: A Human-Centered Pipeline to Streamline Systematic Literature Reviews with Large Language Models,Joao Torres,4,Information,8,"The growing volume of academic publications poses significant challenges for researchers conducting timely and accurate systematic literature reviews (SLR), particularly in fast-evolving fields like artificial intelligence. This growth of academic literature also makes it increasingly difficult for lay people to access scientific knowledge effectively, meaning academic literature is often misrepresented in the popular press and, more broadly, in society. Traditional SLRs are labor-intensive and error-prone, and they struggle to keep up with the rapid pace of new research. To address these issues, we developed PROMPTHEUS: an AI-driven pipeline solution that automates the systematic literature review (LR) process using large language models (LLMs). We aimed to enhance efficiency by reducing the manual workload while maintaining the precision and coherence required for comprehensive literature synthesis. PROMPTHEUS automates key SLR stages, including systematic searches, data extraction, and topic modeling using BERTopic and summarization with transformer models. Evaluations across five research domains demonstrated that PROMPTHEUS reduces review time, achieves high precision, and provides coherent topic organization, offering a scalable and effective solution for conducting literature reviews in an increasingly crowded research landscape." +W7202271496,,10.1002/lary.70801,2026,ENTGPT : Applying Large Language Models to Systematic Review Screening With the Novel STARR Protocol,Akash Kapoor,6,The Laryngoscope,0,"OBJECTIVES: Systematic literature reviews (SLRs) are time-intensive and resource-consuming. While large language models (LLMs) have shown promise in encoding clinical knowledge, evidence for their performance on complex text analysis, necessary for SLRs in otolaryngology, remains limited. In this proof-of-concept study, we investigate an LLM's performance in screening relevant articles for a SLR with the novel screening of title and abstracts, reevaluation, and full-text review (STARR) protocol. METHODS: ENTGPT (based on GPT-4o) was compared to two human reviewers in article inclusion/exclusion decisions using the traditional and STARR screening protocols. ENTGPT was provided with inclusion and exclusion criteria, titles, abstracts, and full texts (if available) of the 850 articles retrieved in the original search. The model's decisions were compared to those made by two human reviewers. RESULTS: ENTGPT, using the STARR protocol, achieved 99.87% accuracy in article classification compared with human reviewers (95% CI: 0.99-1.0), including 100% specificity and 95% sensitivity. When using the traditional protocol, sensitivity declined to 35%. ENTGPT, using the traditional protocol, achieved 99.47% accuracy in article classification compared with human reviewers (95% CI: 0.99-1.0), including 100% specificity and 35% sensitivity. When using the STARR protocol, accuracy improved to 99.87% and sensitivity markedly increased to 95%. CONCLUSIONS: ENTGPT accurately replicated human reviewers in article selection and data extraction for an otolaryngology SLR using the STARR and traditional protocols. This performance suggests that LLMs could be employed to significantly streamline the SLR process, potentially saving substantial time and resources for researchers. LEVEL OF EVIDENCE: N/A." +W7166589016,,10.63317/5dnbkz4d44kz,2026,Automatic Generation of Discharge Summaries Using Large Language Models: A Systematic Literature Review,Lucas Molino-Piñar,3,Language Resources and Evaluation Conference (LREC),0,"Discharge summaries are critical documents for continuity of care, yet their manual creation imposes significant burdens on clinical staff.This systematic literature review examines current approaches to automatic generation of discharge summaries using Natural Language Processing (NLP) and Large Language Models (LLMs).Following the Kitchenham guidelines for systematic reviews in software engineering, we searched Scopus and PubMed databases for studies published between 2023 and 2026, identifying 9 primary studies from an initial pool of 102 papers.Our analysis reveals that GPT-4 and its variants dominate current research (appearing in 6 of 9 studies), while open-source alternatives like LLaMA show promise for privacy-preserving deployments.Evaluation primarily relies on automatic metrics (ROUGE, BLEU) combined with human expert assessment.Key challenges include hallucination rates ranging from 33% to 64%, information omission, integration with Electronic Health Record (EHR) systems, and context window limitations.Studies addressing factuality employ human-in-the-loop validation, prompt engineering techniques, and knowledge graph-based correction mechanisms.Despite these challenges, recent implementations demonstrate clinical feasibility, with one study achieving a 94.35% System Usability Score.This review provides a comprehensive synthesis of the state-of-the-art and identifies opportunities for future research in this rapidly evolving field." +W7160102024,,10.1017/rsm.2026.10124,2026,MechaScreener: Large language model-based automated screening for systematic reviews and research,Connor Forbes,5,Research Synthesis Methods,0,"Systematic reviews are the gold standard for evidence synthesis, but the manual title and abstract screening of thousands of references creates a severe bottleneck. Existing automated tools have historically struggled to achieve the near-perfect recall (sensitivity) required for reliable reviews. We developed MechaScreener as a 'zero-shot' automated screening tool that utilises a large language model to rank article relevance. The tool requires no initial training data or manual pre-screening, as MechaScreener directly applies user-provided question elements (PICO) or inclusion/exclusion criteria to assign an ordinal relevance score (1-5) to each reference. We evaluated the tool in two phases: a development phase using five reference libraries to optimise prompts, and an independent evaluation phase using 10 diverse Cochrane review libraries (comprising both randomised controlled trials [RCTs] and non-RCTs) containing over 58,000 references. In the evaluation dataset, MechaScreener achieved a perfect recall of 1.00 (100%) in all 10 reviews, ensuring that no relevant articles were missed (pooled record-level 95% CI: 0.98-1.00). Concurrently, it achieved an overall macro-average specificity of 0.61 (61%, cluster bootstrap 95% CI: 0.46-0.75; pooled record-level specificity 0.59, 95% CI: 0.59-0.60). Specificity varied from 0.21 in broad public health topics to 0.91 in precise pharmacological interventions, reflecting the tool's built-in conservatism when evaluating ambiguous abstracts. By safely eliminating over 60% of irrelevant literature during the initial screening phase without compromising recall, MechaScreener functions as a highly reliable but low-effort 'first-pass' filter, allowing researchers to substantially reduce manual workloads and reallocate resources towards full-text review and data extraction." +W7140887005,,10.1145/3802827,2026,Enhancing Automated Unit Test Generation with Large Language Models: A Systematic Literature Review,Junwei Zhang,5,ACM Transactions on Software Engineering and Methodology,4,"Automated unit test generation is a fundamental yet challenging task in software engineering (SE), playing a critical role in ensuring software correctness, reliability, and maintainability. While traditional approaches such as search-based software testing and symbolic execution have achieved notable success, they often suffer from limited semantic understanding, high configuration costs, and scalability constraints. Recent advances in Large Language Models (LLMs) have fundamentally reshaped the landscape of automated unit testing by enabling models to reason over source code semantics and generate executable, context-aware test cases. Despite the rapid growth of this research area, a comprehensive and task-oriented synthesis of existing work remains lacking. This article presents a systematic literature review of LLM-based unit test generation. This review draws on research from leading SE and AI conferences and journals, including 69 papers published across 25 distinct venues, along with 47 high-quality preprint papers, bringing the total to 116. Our review aims to answer three key research questions: (1) which unit testing tasks have been addressed using LLMs, (2) how LLMs are adapted and integrated into the unit test generation pipeline, and (3) what datasets, benchmarks, and evaluation practices are employed in existing studies. To this end, we organize the literature from a task-centric perspective, covering test generation, test input generation, test oracle generation, and test evolution, and from a methodological perspective, categorizing LLM adaptation strategies into fine-tuning, prompt engineering, and agent-based approaches. Our analysis reveals that current research predominantly focuses on function- and class-level test generation, with comparatively limited attention given to test input generation, oracle construction, and long-term test evolution. Decoder-only LLMs, particularly GPT-family and LLaMA-based models, dominate the field, while encoder-only and encoder–decoder models remain underexplored. We further observe substantial disparities in dataset characteristics, programming language coverage, and evaluation metrics, which hinder fair comparison and reproducibility across studies. Based on empirical evidence extracted from the surveyed literature, we identify key challenges facing LLM-based unit test generation. Building on these findings, we outline several promising research directions, such as dataset optimization, structure-aware context modeling, agent coordination mechanisms, and benchmark enhancement. This review provides a consolidated and evidence-driven foundation for future research, aiming to advance the development of scalable, reliable, and practically applicable LLM-driven unit testing techniques." +W7147185311,,10.1109/iccsm66818.2025.00027,2025,Toward the Conduction of Automated Systematic Literature Reviews: The Role of Large Language Models,Marco Fiore,1,"International Conference on Computer, Software and Modeling (ICCSM)",0,"The conduction of Systematic Literature Reviews is an essential yet labor-intensive process for researchers, requiring meticulous compilation and analysis of literature to answer predefined Research Questions. Despite attempts to streamline this process, automation of reviews remains a significant challenge. This paper explores the potential of Generative AI and Large Language Models to revolutionize the review process by automating key tasks such as identifying, extracting, and synthesizing relevant literature. By leveraging open-source Large Language Models, we aim to reduce the time and effort traditionally required for comprehensive literature reviews. Our study involves comparing various models, including lightweight and more robust ones. We replicate a published review on Blockchain technology in agri-food traceability to evaluate the performance of these models. Results indicate that while high-parameter models successfully generate relevant search strings and classifications, they also exhibit trade-offs in terms of response times. Our findings highlight both the promise and limitations of current Large Language Models in automating Systematic Literature Reviews." +W7155247769,,,2026,Large Language Models in Factory Planning: A Systematic Literature Review of an Emerging Research Field,Richard Reider,1,OSF Preprints (OSF Preprints),0,"This project is a systematic literature review of large language models (LLMs) and agentic AI systems applied to factory planning as defined by VDI 5200 and the framework of Wiendahl, Reichardt & Nyhuis — covering strategic, concept, and detail planning phases from requirements elicitation to layout design. Motivated by the rapid emergence of LLM-based assistance systems in manufacturing since 2023 and the absence of a dedicated synthesis at the factory-planning level, the review asks what tasks LLMs currently address, which architectural patterns (RAG, multi-agent systems, LLM+solver/digital-twin hybrids) have emerged, what evidence exists on effectiveness, and where the research gaps lie. The search follows PRISMA 2020 and combines a database track (Scopus TITLE-ABS-KEY, Web of Science TS=, and OpenAlex full-text) with a parallel hand-search track covering German grey literature (Fraunhofer Publica, ZWF archive, mediaTUM, KLUEDO) and seed-paper snowballing from anchor works by Ferry/Huang/Reinhart (ZWF 2025), Klar et al. (JMS 2024), and the Fraunhofer GenAI4FFD project. Screening uses dual independent review on a 10% sample with Cohen's κ ≥ 0.7 before single-rater continuation, and a tiered quality rubric weights venue rank, methodological transparency, and empirical grounding during synthesis. All materials — protocol, search strings, screening logs, extraction tables, and PRISMA flow diagram — are shared openly to ensure reproducibility and to support living-review updates as the field matures." +W4415271247,,10.1007/s10462-025-11398-1,2025,A systematic review of multi-modal large language models on domain-specific applications,Sirui Li,4,Artificial Intelligence Review,19,"Abstract While Large Language Models (LLMs) have shown remarkable proficiency in text-based tasks, they struggle to interact effectively with the more realistic world without the perceptions of other modalities such as visual and audio. Multi-modal LLMs, which integrate these additional modalities, have become increasingly important across various domains. Despite the significant advancements and potential of multi-modal LLMs, there has been no comprehensive PRISMA-based systematic review that examines their applications across different domains. The objective of this work is to fill this gap by systematically reviewing and synthesising the quantitative research literature on domain-specific applications of multi-modal LLMs. This systematic review follows the PRISMA guidelines to analyse research literature published after 2022, the release of OpenAI’s ChatGPT $$-$$ 3.5. The literature search was conducted across several online databases, including Nature, Scopus, and Google Scholar. A total of 22 studies were identified, with 11 focusing on the medical domain, 3 on autonomous driving, and 2 on geometric analysis. The remaining studies covered a range of topics, with one each on climate, music, e-commerce, sentiment analysis, human-robot interaction, and construction. This review provides a comprehensive overview of the current state of multi-modal LLMs, highlights their domain-specific applications, and identifies gaps and future research directions." +W4401633394,,10.2196/52758,2024,Human-Comparable Sensitivity of Large Language Models in Identifying Eligible Studies Through Title and Abstract Screening: 3-Layer Strategy Using GPT-3.5 and GPT-4 for Systematic Reviews,Kentaro Matsui,6,Journal of Medical Internet Research,50,"BACKGROUND: The screening process for systematic reviews is resource-intensive. Although previous machine learning solutions have reported reductions in workload, they risked excluding relevant papers. OBJECTIVE: We evaluated the performance of a 3-layer screening method using GPT-3.5 and GPT-4 to streamline the title and abstract-screening process for systematic reviews. Our goal is to develop a screening method that maximizes sensitivity for identifying relevant records. METHODS: We conducted screenings on 2 of our previous systematic reviews related to the treatment of bipolar disorder, with 1381 records from the first review and 3146 from the second. Screenings were conducted using GPT-3.5 (gpt-3.5-turbo-0125) and GPT-4 (gpt-4-0125-preview) across three layers: (1) research design, (2) target patients, and (3) interventions and controls. The 3-layer screening was conducted using prompts tailored to each study. During this process, information extraction according to each study's inclusion criteria and optimization for screening were carried out using a GPT-4-based flow without manual adjustments. Records were evaluated at each layer, and those meeting the inclusion criteria at all layers were subsequently judged as included. RESULTS: On each layer, both GPT-3.5 and GPT-4 were able to process about 110 records per minute, and the total time required for screening the first and second studies was approximately 1 hour and 2 hours, respectively. In the first study, the sensitivities/specificities of the GPT-3.5 and GPT-4 were 0.900/0.709 and 0.806/0.996, respectively. Both screenings by GPT-3.5 and GPT-4 judged all 6 records used for the meta-analysis as included. In the second study, the sensitivities/specificities of the GPT-3.5 and GPT-4 were 0.958/0.116 and 0.875/0.855, respectively. The sensitivities for the relevant records align with those of human evaluators: 0.867-1.000 for the first study and 0.776-0.979 for the second study. Both screenings by GPT-3.5 and GPT-4 judged all 9 records used for the meta-analysis as included. After accounting for justifiably excluded records by GPT-4, the sensitivities/specificities of the GPT-4 screening were 0.962/0.996 in the first study and 0.943/0.855 in the second study. Further investigation indicated that the cases incorrectly excluded by GPT-3.5 were due to a lack of domain knowledge, while the cases incorrectly excluded by GPT-4 were due to misinterpretations of the inclusion criteria. CONCLUSIONS: Our 3-layer screening method with GPT-4 demonstrated acceptable level of sensitivity and specificity that supports its practical application in systematic review screenings. Future research should aim to generalize this approach and explore its effectiveness in diverse settings, both medical and nonmedical, to fully establish its use and operational feasibility." +W4415949686,,10.1017/rsm.2026.10110,2026,Evaluating prompting strategies and large language models in systematic literature review screening: Relevance and task-stage classification,Binglan Han,4,Research Synthesis Methods,0,"This study quantifies how prompting strategies interact with large language models (LLMs) to automate the screening stage of systematic literature reviews (SLRs). We evaluate six LLMs (GPT-4o, GPT-4o-mini, DeepSeek-Chat-V3, Gemini-2.5-Flash, Claude-3.5-Haiku, and Llama-4-Maverick) under five prompt types (zero-shot, few-shot, chain-of-thought [CoT], CoT-few-shot, and self-reflection) across relevance classification and six Level-2 tasks, using accuracy, precision, recall, and F1. Results show pronounced model-prompt interaction effects within this SLR-automation corpus: zero-shot maximizes recall for high-sensitivity relevance screening; CoT-few-shot yields stronger precision and F1; and self-reflection underperforms due to over-inclusivity and instability across models. GPT-4o and DeepSeek provide robust overall performance, while GPT-4o-mini performs competitively at a substantially lower dollar cost. A cost-performance analysis for relevance classification (per 1,000 abstracts) reveals large absolute differences among model-prompt pairings; GPT-4o-Mini remains low-cost across prompts, and structured prompts (CoT/CoT-few-shot) on GPT-4o-Mini offer attractive F1 at a small incremental cost. The cost-performance analysis also suggests that future screening workflows could incorporate staged, cost-aware procedures: (1) using lower-cost, high-recall configurations for initial screening and (2) reserving higher-cost or more precision-oriented configurations for later triage or uncertain records. These findings highlight LLMs' uneven but promising potential to support screening within corpora similar to the SLR-automation corpus examined in this study. Further validation is needed before generalising these results to broader systematic review screening tasks. By systematically analyzing prompt-model interactions, we provide a comparative benchmark and practical guidance for task-adaptive LLM deployment." +W4417081257,,10.1007/s10758-025-09922-x,2025,Large Language Model Applications in the Algebra Domain: A Systematic Review,Yajie Song,5,Technology Knowledge and Learning,0, +W4409794823,,10.2196/64963,2025,Comparing Diagnostic Accuracy of Clinical Professionals and Large Language Models: Systematic Review and Meta-Analysis,Guxue Shan,7,JMIR Medical Informatics,71,"Background: With the rapid development of artificial intelligence (AI) technology, especially generative AI, large language models (LLMs) have shown great potential in the medical field. Through massive medical data training, it can understand complex medical texts and can quickly analyze medical records and provide health counseling and diagnostic advice directly, especially in rare diseases. However, no study has yet compared and extensively discussed the diagnostic performance of LLMs with that of physicians. Objective: This study systematically reviewed the accuracy of LLMs in clinical diagnosis and provided reference for further clinical application. Methods: We conducted searches in CNKI (China National Knowledge Infrastructure), VIP Database, SinoMed, PubMed, Web of Science, Embase, and CINAHL (Cumulative Index to Nursing and Allied Health Literature) from January 1, 2017, to the present. A total of 2 reviewers independently screened the literature and extracted relevant information. The risk of bias was assessed using the Prediction Model Risk of Bias Assessment Tool (PROBAST), which evaluates both the risk of bias and the applicability of included studies. Results: A total of 30 studies involving 19 LLMs and a total of 4762 cases were included. The quality assessment indicated a high risk of bias in the majority of studies, primary cause is known case diagnosis. For the optimal model, the accuracy of the primary diagnosis ranged from 25% to 97.8%, while the triage accuracy ranged from 66.5% to 98%. Conclusions: LLMs have demonstrated considerable diagnostic capabilities and significant potential for application across various clinical cases. Although their accuracy still falls short of that of clinical professionals, if used cautiously, they have the potential to become one of the best intelligent assistants in the field of human health care." +W7116974613,,10.2196/preprints.89862,2025,Large Language Models in Colorectal Cancer: A Systematic Review (Preprint),Jinglei Tian,6,,0,"UNSTRUCTURED This systematic review aimed to analyze the application and efficacy of large language models (LLMs) in the full-cycle management of colorectal cancer (CRC). CRC diagnosis and treatment face challenges in multi-stage collaboration and integration of large-scale clinical data. While LLMs offer potential solutions, targeted evidence in CRC-specific settings remains insufficient. Following PRISMA guidelines, six databases including PubMed, Embase, and Web of Science were systematically searched up to November 1, 2025. Two reviewers independently screened studies, extracted data, and assessed quality using appropriate tools (QUADAS-2, PROBAST, ROBINS-I) based on study design. A total of 34 studies were included. LLMs were primarily applied in auxiliary diagnosis, information extraction, knowledge-based question answering, treatment decision support, and predictive modeling. GPT-4 achieved a diagnostic accuracy of 97.2% and an F1-score of 84.1% in information extraction, while lightweight or domain-specialized models (e.g., Gemma-2, RoBERTa) performed optimally in specific tasks. Significant heterogeneity was observed across studies due to variations in model types, prompt engineering strategies, and outcome measures. In conclusion, LLMs demonstrate considerable potential in supporting CRC management, particularly in data structuring and decision-making. However, limitations include data bias, model hallucination, and insufficient generalizability. Future research should focus on multicenter prospective validation, improvement of model interpretability, and development of multimodal and domain-adapted LLMs for safe and effective clinical integration." +W4408680354,,10.2196/preprints.74177,2025,Large Language Models in Lung Cancer: A Systematic Review (Preprint),Ruikang Zhong,9,,0,"BACKGROUND In the era of data and intelligence, Artificial Intelligence (AI), especially Large Language Model (LLM), has been widely applied in the medical field. OBJECTIVE The aim of this systematic review is to discuss the progress of LLM application in lung cancer, including clinical, educational and research aspects, and to explore the potential of LLM application in the full-cycle management of lung cancer. METHODS We searched six electronic databases according to PRISMA guidelines. Information was screened and extracted independently by two authors according to the inclusion criteria. Quality assessment was performed using the QUADAS-2, PROBAST and ROBINS-I tools. RESULTS The literature search yielded 706 relevant studies, resulting in the inclusion of a total of 28 studies published between 2023 and 2024.LLMs are widely used in lung cancer-related tasks, including assisted diagnosis, information extraction, question and answer science, and therapeutic decision support.ChatGPT is the most commonly used model and shows significant potential for improving diagnostic accuracy and patient communication. The importance of cue engineering and fine-tuning in optimising the performance of LLMs in specific clinical tasks is also highlighted. CONCLUSIONS LLMs have the potential to transform lung cancer management by improving diagnostic accuracy, patient communication and treatment planning. However, issues such as data security, ethical regulations, economic costs and clinical validation need to be addressed and LLMs need to be used rationally as effective tools rather than misused or even replaced by healthcare professionals." +W7135056671,,10.1109/sami68106.2026.11420686,2026,Assessing Probability Rating Performance for Large Language Models in Systematic Literature Review Automation,Márk Zrubka,3,IEEE World Symposium on Applied Machine Intelligence and Informatics (SAMI),1,"Classical machine learning methods of systematic literature review (SLR) automation use thresholded probability scores to make decisions about including or excluding articles in the title/abstract selection process. Large language models (LLMs) have gained popularity in SLR automation and are sometimes prompted to give similar probability score outputs as classical models. LLMs do not always perform well in mathematical reasoning tasks, which raises the question whether their probability outputs are meaningful. We evaluated the accuracy of four LLMs in title/abstract screening, using qualitative rating (Yes/Unsure/No) and a continuous probability rating (0-1). We also assessed the consistency between probability and qualitative ratings within each model using violin plots. The results show that LLMs performed well in excluding irrelevant articles, but their performance dropped notably for including relevant articles. Claude Sonnet 4 and Phi 4 produced consistent probability and qualitative ratings. GPT 4o-mini did not represent probabilities correctly and Qwen 2.5 underperformed in the qualitative category rating. Due to the varying performance of LLMs, we recommend using multiple LLMs to assist human reviewers in the title/abstract screening process of SLRs. Furthermore, we advise testing whether the output of LLMs is meaningful before using them to evaluate articles." +W4414794728,,10.3233/shti251488,2025,Leveraging ChatGPT-4 for Evidence Synthesis: A Case Study on the Use of a Large Language Model in a Systematic Review,Federica Tomassini,6,Studies in health technology and informatics,0,"Artificial intelligence, particularly Large Language Models (LLM) such as ChatGPT, is emerging as a potentially transformative support for traditionally complex and time-consuming Systematic Literature Reviews (SLRs). In this study, we compared the traditional SLR process executed accordingly with Cochrane guidelines, with an AI-assisted approach using ChatGPT across various stages, from research question formulation to report writing. Effectiveness was assessed through quantitative measurements of time savings at each phase. Results showed substantial time reductions in several operational tasks, including Gantt chart, generating search terms and suggesting selection criteria. However, critical issues arose in stages requiring interpretative judgement, such as analyzing results, assessing risk of bias and final drafting. While AI cannot replace the role of the researcher, it is a valuable tool to optimize SLR workflow. The combination of human expertise and LLM capabilities presents a promising solution, provided it is accompanied by continuous development of AI systems to improve their reliability, transparency and interoperability." +W4399210979,,10.1007/978-3-031-63028-6_6,2024,Preliminary Systematic Review of Open-Source Large Language Models in Education,Michael Pin-Chuan Lin,4,Lecture notes in computer science,12, +W4392863142,,10.3352/jeehp.2024.21.6,2024,"Opportunities, challenges, and future directions of large language models, including ChatGPT in medical education: a systematic scoping review",Xiaojun Xu,3,Journal of Educational Evaluation for Health Professions,165,"BACKGROUND: ChatGPT is a large language model (LLM) based on artificial intelligence (AI) capable of responding in multiple languages and generating nuanced and highly complex responses. While ChatGPT holds promising applications in medical education, its limitations and potential risks cannot be ignored. METHODS: A scoping review was conducted for English articles discussing ChatGPT in the context of medical education published after 2022. A literature search was performed using PubMed/MEDLINE, Embase, and Web of Science databases, and information was extracted from the relevant studies that were ultimately included. RESULTS: ChatGPT exhibits various potential applications in medical education, such as providing personalized learning plans and materials, creating clinical practice simulation scenarios, and assisting in writing articles. However, challenges associated with academic integrity, data accuracy, and potential harm to learning were also highlighted in the literature. The paper emphasizes certain recommendations for using ChatGPT, including the establishment of guidelines. Based on the review, 3 key research areas were proposed: cultivating the ability of medical students to use ChatGPT correctly, integrating ChatGPT into teaching activities and processes, and proposing standards for the use of AI by medical students. CONCLUSION: ChatGPT has the potential to transform medical education, but careful consideration is required for its full integration. To harness the full potential of ChatGPT in medical education, attention should not only be given to the capabilities of AI but also to its impact on students and teachers." +W4415396422,,10.1002/cesm.70042,2025,Enhancing Evidence Synthesis Efficiency: Leveraging Large Language Models and Agentic Workflows for Optimized Literature Screening,Bing Hu,11,Cochrane Evidence Synthesis and Methods,4,"Background: Public health events of international concern highlight the need for up-to-date evidence curated using sustainable processes that are accessible. In development of the Global Repository of Epidemiological Parameters (grEPI) we explore the performance of an agentic-AI assisted pipeline (GREP-Agent) for screening evidence which capitalizes on recent advancements in large language models (LLMs). Methods: In this study, the performance of the GREP-Agent was evaluated on a data set of 2000 citations from a systematic review on measles using four LLMs (GPT4o, GPT4o-mini, Llama3.1, and Phi4). The GREP-Agent framework integrates multiple LLMs and human feedback to fine-tune its performance, optimize workload reduction and accuracy in screening research articles. The impact on performance of each part of this Agentic-AI system is presented and measured by accuracy, precision, recall, and F1-score metrics. Results: The results show how each phase of the GREP-Agent system incrementally improves accuracy regardless of the LLM. We found that GREP-Agent was able to increase sensitivity across a broad range of open source and proprietary LLMs to 84.2%-88.9% after fine-tuning and to 86.4%-95.3% by varying workload reduction strategies. Performance was significantly impacted by the clarity of the screening questions and setting thresholds for optimized workload reduction strategies. Conclusions: The GREP-Agent shows promise in improving the efficiency and effectiveness of evidence synthesis in dynamic public health contexts. Further development and refinement of adaptable human-in-the-loop AI systems for screening literature are essential to support future public health response activities, while maintaining a human-centric approach." +W4411085860,,10.1007/s00405-025-09504-8,2025,Clinical decision support using large language models in otolaryngology: a systematic review,Rania Filali Ansary,2,European Archives of Oto-Rhino-Laryngology,18, +W7135218648,,10.5281/zenodo.18998810,2026,"EXPLORING HALLUCINATIONS IN LARGE LANGUAGE MODELS (LLMs): A SYSTEMATIC REVIEW OF TYPOLOGIES, ORIGINS, AND MITIGATION APPROACHES",AMAL ALTALHI,2,Journal of Tianjin University Science and Technology,0,"Abstract This systematic literature review pull out the understanding of large language models (LLMs) by thoroughly examining hallucination situations, including the types, causes, and reduce approaches to enhance LLM usefulness in natural language processing (NLP). Electronic databases (Web of Science, IEEE Xplore, Open Review, Google Scholar) were queried by a comprehensive search, generating 1136 records. Of these, 27 met the inclusion criteria and were included. A meta-aggregative approach was used to analyze and synthesize the articles. The research questions formed significant themes for organizing the findings and results section. LLM's ordinary taxonomy includes fact hallucination, honesty hallucination, lack of alignment, conflict in ideas, nonsensical hallucination, random hallucination, object hallucination, and intrinsic and external hallucination. Hallucination causes were training data issues, model limitation/ overfitting, limited context window/ knowledge cutoff, and nuanced language understanding. Effective mitigative approaches were domain-specific fine-tuning, prompting, model reprogramming, and grounding." +W4403915968,,10.31219/osf.io/wm2av,2024,Accelerating Systematic Reviews with Large Language Models: Current Practices and Recommendations,Bo HU,2,,3,"This study investigates the application of Large Language Models (LLMs) in systematic reviews, emphasizing their performance, consistency, and potential for cost efficiency. Through a systematic lineage search, we included 76 studies from 734 articles. The findings reveal that LLMs demonstrate moderate to high performance in title and abstract screening, full-text screening, and data extraction. However, their performance is notably unstable in the stages of literature search and quality assessment. Prompt design emerges as a crucial factor, with Chain of Thought (CoT) prompts frequently enhancing results. While LLMs exhibit moderate to high agreement with human reviewers in some stages, their consistency in quality assessment remains comparatively lower. The research suggests that while LLMs cannot fully replace human reviewers, they serve as valuable assistants in systematic reviews, especially in reducing time and effort. The study also provides practical recommendations for integrating LLMs effectively and discusses the challenges and future research directions in this evolving field." +W7171408732,,10.15173/mumj.v22i2.4166,2026,Performance of large language models on screening titles and abstracts of urology-related systematic reviews,Ata B. Gungor,4,McMaster University Medical Journal,0,"Background: Systematic reviews summarize research evidence to inform clinical practice guidelines and health policy, but the review process is labour-intensive. Authors often screen thousands of abstracts to identify relevant studies to include in their review. Large Language Models (LLMs) may improve title and abstract screening efficiency, but their performance compared to humans require evaluation. Objective: To assess whether locally deployed LLMs accurately screen titles and abstracts of urology-related systematic reviews. Methods: We used LLMs to screen titles and abstracts from three Cochrane reviews in urology totaling 8020 records. Records were screened using predefined criteria, as stated in the Cochrane manuscripts, using three strategies: Qwen3 30B A3B Instruct 2507 alone, Ministral 3 14B Instruct 2512 alone, and their ensemble. A “first-ahead-by-k” approach (k = 2) was used with two models in parallel. We compared their outputs, and once a classification led by a margin of K (i.e., it appeared two more times than any alternative in the running tally) we selected that label as the consensus. Strategy performance metrics were calculated against the human-defined gold standard. Results: Specificity consistently exceeded 90% across strategies (Figure 1). Pooled estimates of sensitivity across the three reviews for strategies were: Ministral 3 14B, 71.1% (95% CI, 64.9-76.6); Qwen3 30B A3B, 53.1% (46.6-59.4); and their ensemble, 72.2% (66.1-77.7). Conclusion: Locally deployed LLMs can screen titles and abstracts with moderate accuracy. Next steps to improve LLM performance include creating random subsamples from each systematic review to perform calibration screening runs with the models, reviewing discrepancies between the model and human-defined gold standard to determine why the performance was variable, and retrieving the exact title and abstract screening criteria used by the authors." +W4413051957,,10.1038/s41746-025-01840-7,2025,Accelerating clinical evidence synthesis with large language models,Zifeng Wang,6,npj Digital Medicine,68,"Clinical evidence synthesis largely relies on systematic reviews (SR) of clinical studies from medical literature. Here, we propose a generative artificial intelligence (AI) pipeline named TrialMind to streamline study search, study screening, and data extraction tasks in SR. We chose published SRs to build TrialReviewBench, which contains 100 SRs and 2,220 clinical studies. For study search, it achieves high recall rates (Ours 0.711-0.834 v.s. Human baseline 0.138-0.232). For study screening, TrialMind beats previous document ranking methods in a 1.5-2.6 fold change. For data extraction, it outperforms a GPT-4's accuracy by 16-32%. In a pilot study, human-AI collaboration with TrialMind improved recall by 71.4% and reduced screening time by 44.2%, while in data extraction, accuracy increased by 23.5% with a 63.4% time reduction. Medical experts preferred TrialMind's synthesized evidence over GPT-4's in 62.5%-100% of cases. These findings show the promise of accelerating clinical evidence synthesis driven by human-AI collaboration." +W7167031171,,10.3233/shti260813,2026,Exploratory Evaluation of Large Language Models for Reducing Language Bias in Systematic Review Screening,Junki Ikeguchi,3,Studies in health technology and informatics,0,"Language bias arises in systematic reviews when non-English studies are excluded owing to resource constraints. Large language models (LLMs) can mitigate this problem through multilingual processing. To assess whether direct multilingual LLM processing reduces language-based disparities in systematic review screening performance compared to translation-mediated approaches. Six state-of-the-art LLMs were evaluated under three conditions: (1) an English benchmark dataset (n = 2,911), (2) direct screening of non-English abstracts (n = 483), and (3) screening of machine-translated non-English abstracts. Performance was measured using sensitivity, specificity, F1 score, balanced accuracy, and workload reduction. All models achieved high sensitivity on English data (≥0.938). Translation-mediated screening substantially reduced sensitivity in some models (range: 0.47-0.54), whereas direct multilingual processing maintained high sensitivity (range: 0.71-1.00). Considerable differences were observed among models. Direct multilingual LLM screening may reduce language-related sensitivity disparities; however, the effects on downstream meta-analytic bias require further investigation." +W7218139948,,,2025,Analysis of the Effectiveness of Large Language Models in Industrial Knowledge Management: A Systematic Literature Review,Jeronimo Malaquias,1,Jurnal Mekintek Jurnal Mekanikal Energi Industri Dan Teknologi,0,"The increasing adoption of Large Language Models (LLMs) has transformed the way organizations manage, access, and utilize knowledge in industrial environments. As industries continue to generate vast amounts of information, LLMs have emerged as powerful tools for enhancing knowledge management processes through advanced natural language understanding, information retrieval, and intelligent decision support capabilities. This study aims to analyze the effectiveness of LLMs in supporting industrial knowledge management and to evaluate their contributions, benefits, and challenges across organizational contexts. A Systematic Literature Review (SLR) approach was employed to examine relevant studies published between 2020 and 2026, with data collected from major scientific databases, including Scopus, Web of Science, IEEE Xplore, and ScienceDirect. The selected literature was analyzed using thematic, content, and comparative analysis techniques to identify patterns, applications, and implementation outcomes. The findings indicate that LLMs significantly enhance knowledge creation through automated report generation, technical documentation support, and lessons-learned extraction. Furthermore, LLM adoption contributes to increased organizational productivity by reducing information search time, supporting decision-making, and improving employee access to critical knowledge resources. However, several challenges remain, including hallucination, data inconsistency, model bias, integration complexity, security and privacy concerns, and issues related to transparency, accountability, and explainability. To maximize their benefits, organizations should implement robust AI governance frameworks, adopt secure knowledge retrieval architectures, and invest in employee AI literacy and training programs. Future research should focus on real-world industrial evaluations, comparative analyses of LLM platforms, and long-term assessments of organizational impacts." +W4403973695,,10.1101/2024.10.30.24316199,2024,Clinical Applications and Limitations of Large Language Models in Nephrology: A Systematic Review,Zoe Unger,6,medRxiv,2,"Abstract Background Large Language Models (LLMs) are emerging as promising tools in healthcare. This systematic review examines LLMs’ potential applications in nephrology, highlighting their benefits and limitations. Methods We conducted a literature search in PubMed and Web of Science, selecting studies based on Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) guidelines. The review focuses on the latest advancements of LLMs in nephrology from 2020 to 2024. PROSPERO registration number: CRD42024550169. Results Fourteen studies met the inclusion criteria and were categorized into five key areas of nephrology: Streamlining workflow, disease prediction and prognosis, laboratory data interpretation and management, renal dietary management, and patient education. LLMs showed high performance in various clinical tasks, including managing continuous renal replacement therapy (CRRT) alarms (GPT-4 accuracy 90-94%) for reducing intensive care unit (ICU) alarm fatigue, and predicting chronic kidney diseases (CKD) progression (improved positive predictive value from 6.7% to 20.9%). In patient education, GPT-4 excelled at simplifying medical information by reducing readability complexity, and accurately translating kidney transplant resources. Gemini provided the most accurate responses to frequently asked questions (FAQs) about CKD. Conclusions While the incorporation of LLMs in nephrology shows promise across various levels of patient care, their broad implementation is still premature. Further research is required to validate these tools in terms of accuracy, rare and critical conditions, and real-world performance." +W7214983381,,10.1016/j.jss.2026.113129,2026,Large language models in software project management: A two-phase systematic literature review,Muhammad Saad Amin,4,Journal of Systems and Software,0,"Background Large Language Models (LLMs) are increasingly integrated into Software Engineering (SE) processes and practices. However, their application within Software Project Management (SPM) remains fragmented and insufficiently synthesized. This absence of systematic evidence prevents both researchers and practitioners from understanding where LLMs can meaningfully contribute to project management processes in relation to established standards such as the Project Management Body of Knowledge (PMBOK). Objectives This study aims to systematically synthesize existing evidence on the application and integration of LLMs in SPM by means of a two-phase method where Phase I systematically searches for existing secondary studies in the topic of interest, and Phase II (whenever applicable) carries out an SLR in the topic of interest. Method We conducted a two-phase method: Phase I is a tertiary study that used 14 research questions to assess the existence, coverage, and quality of prior secondary studies in LLMs applied to SPM. As no suitable SLR was identified from Phase I, we also conducted Phase II, which employed the SEGRESS guidelines to analyze primary studies that empirically evaluated the use of LLMs in SPM via five research questions. Phase II also assessed coverage across PMBOK processes and synthesized reported mechanisms, outcomes, research designs and evaluation approaches. Results The SLR (Phase II) shows that studies employ encoder-only and decoder-only LLM architectures in equal proportion (17 studies each). Their applications focus on a narrow set of SPM tasks, including effort estimation, project planning, and requirements-related activities. Coverage across PMBOK processes is uneven, with limited attention to areas such as procurement, stakeholder, quality, and cost management. Most studies rely on proof-of-concept evaluations, with scarce industrial validation and heterogeneous evaluation metrics. Conclusion The results indicate that current research on LLMs in SPM provides limited guidance for systematic adoption. One of the main drawbacks for such lack of guidance is the weak empirical validation in many of the included primary studies, limited PMBOK coverage, and lack of context-aware integration of LLMs in SPM. 1" +W7136439950,,10.1186/s12873-026-01521-y,2026,Accuracy of the large language model ChatGPT in adult emergency department triage: a systematic review and meta-analysis,Shuang Gao,6,BMC Emergency Medicine,3,"AIM: This study employs a systematic review and meta-analysis to evaluate the accuracy and sensitivity (i.e., correct identification rate of high-risk patients at levels 1–2) of ChatGPT in adult emergency triage and to compare the potential of its different versions. METHODS: A systematic search of English and Chinese medical databases (2022–2025) was performed to identify studies directly comparing the triage performance of ChatGPT with that of human experts or a gold standard. The included studies comprised both simulated case scenarios and real-world clinical data from emergency departments. Literature screening adhered to the PICO framework. Methodological quality was evaluated using the QUADAS-2 tool. Data on overall accuracy and sensitivity were extracted. Meta-analysis was conducted in RevMan 5.4, with a predefined subgroup analysis based on the ChatGPT version. RESULTS: A total of 15 studies were included. The pooled accuracy was 0.51 (95% CI: 0.48–0.55) for ChatGPT-3.5, 0.70 (95% CI: 0.58–0.81) for ChatGPT-4 and its variants, and 0.81 for the optimized versions of ChatGPT (with the 95% confidence interval upper bound approaching 1.0 due to high heterogeneity). No significant differences were observed between ChatGPT and human triage in overall accuracy or sensitivity overall. Subgroup analyses showed that ChatGPT-3.5 performed similarly to humans on both metrics. In contrast, ChatGPT-4 and its variants demonstrated higher accuracy (RR = 1.04, P < 0.001) and sensitivity (RR = 1.46, P = 0.04) compared to human triage, but these advantages were not robust-sensitivity analyses revealed that the statistical significance depended on the inclusion or exclusion of specific studies. CONCLUSION: ChatGPT, particularly ChatGPT-4 and its variants, shows promising capability in identifying high-risk cases during triage, and improvements in version and model optimization are associated with enhanced performance. Nevertheless, this finding is not robust, and the evidence supporting the superiority of ChatGPT-4 remains fragile. Consequently, clinical application should be approached with caution and tailored to specific contexts, with ChatGPT serving as an auxiliary tool." +W4412432132,,10.1016/j.jval.2025.04.1191,2025,MSR39 Large Language Model and Contextual Prioritization of Titles and Abstracts for Systematic Review Screening,Riaz Qureshi,2,Value in Health,0, +W4410251088,,10.1186/s12874-025-02583-5,2025,Streamlining systematic reviews with large language models using prompt engineering and retrieval augmented generation,Fouad S. Trad,7,BMC Medical Research Methodology,43,"BACKGROUND: Systematic reviews (SRs) are essential to formulate evidence-based guidelines but require time-consuming and costly literature screening. Large Language Models (LLMs) can be a powerful tool to expedite SRs. METHODS: We conducted a comparative study to evaluate the performance of a commercial tool, Rayyan, and an in-house LLM-based system in automating the screening of a completed SR on Vitamin D and falls. The SR retrieved 14,439 articles, and Rayyan was trained with 2,000 manually screened articles to categorize the rest as most likely to exclude/include, likely to exclude/include and undecided. We analyzed Rayyan's title/abstract screening performance using different inclusion thresholds. For the LLM, we used prompt engineering for title/abstract screening and Retrieval-Augmented Generation (RAG) for full-text screening. We evaluated performance using article exclusion rate (AER), false negative rate (FNR), specificity, positive predictive value (PPV), and negative predictive value (NPV). Additionally, we compared the time required to complete screening steps of the SR using both approaches against the manual screening method. RESULTS: Using Rayyan, including considered as undecided or likely to include for title/abstract screening resulted in an AER of 72.1% and an FNR of 5%. The total estimated screening time, including manual review of articles flagged by Rayyan, was 54.7 hours. Lowering the Rayyan threshold to 'likely to exclude' reduced the FNR to 0% and the AER to 50.7%, but increased the screening time to 81.3 h. Using the LLM system, after title/abstract and full-text screening, 78 articles remained for manual review, including all 20 identified by traditional methods. The LLM achieved an AER of 99.5%, specificity of 99.6%, PPV of 25.6%, and NPV of 100%, with a total screening time of 25.5 h, including manual review of the 78 articles, reducing the manual screening time by 95.5%. CONCLUSIONS: The LLM-based system significantly enhances SR efficiency, compared to manual methods and Rayyan while maintaining low FNR." +W4415757150,,10.2139/ssrn.5688905,2025,Systematic Review of Large Language Model Applications in Transport Engineering,Sneharsh Belsare,4,SSRN Electronic Journal,0, +W4417479740,,10.1016/j.jval.2025.09.2960,2025,SA18 Calibrating Large Language Model Probabilities to Achieve Target Recall in Systematic Review Screening,Ṣẹ̀yẹ Abògúnr̀in,3,Value in Health,0, +W4406849765,,10.3389/fdata.2024.1505284,2025,On explaining recommendations with Large Language Models: a review,Alan Said,1,Frontiers in Big Data,34,"The rise of Large Language Models (LLMs), such as LLaMA and ChatGPT, has opened new opportunities for enhancing recommender systems through improved explainability. This paper provides a systematic literature review focused on leveraging LLMs to generate explanations for recommendations-a critical aspect for fostering transparency and user trust. We conducted a comprehensive search within the ACM Guide to Computing Literature, covering publications from the launch of ChatGPT (November 2022) to the present (November 2024). Our search yielded 232 articles, but after applying inclusion criteria, only six were identified as directly addressing the use of LLMs in explaining recommendations. This scarcity highlights that, despite the rise of LLMs, their application in explainable recommender systems is still in an early stage. We analyze these select studies to understand current methodologies, identify challenges, and suggest directions for future research. Our findings underscore the potential of LLMs improving explanations of recommender systems and encourage the development of more transparent and user-centric recommendation explanation solutions." +W7125684565,,10.2139/ssrn.6086099,2026,Systematic Review of Large Language Model Applications in Transport Engineering,Ashish Bhaskar,4,SSRN Electronic Journal,0, +W4406421570,,10.1093/jamia/ocaf008,2025,"Improving large language model applications in biomedicine with retrieval-augmented generation: a systematic review, meta-analysis, and clinical development guidelines",Siru Liu,3,Journal of the American Medical Informatics Association,197,"OBJECTIVE: The objectives of this study are to synthesize findings from recent research of retrieval-augmented generation (RAG) and large language models (LLMs) in biomedicine and provide clinical development guidelines to improve effectiveness. MATERIALS AND METHODS: We conducted a systematic literature review and a meta-analysis. The report was created in adherence to the Preferred Reporting Items for Systematic Reviews and Meta-Analyses 2020 analysis. Searches were performed in 3 databases (PubMed, Embase, PsycINFO) using terms related to ""retrieval augmented generation"" and ""large language model,"" for articles published in 2023 and 2024. We selected studies that compared baseline LLM performance with RAG performance. We developed a random-effect meta-analysis model, using odds ratio as the effect size. RESULTS: Among 335 studies, 20 were included in this literature review. The pooled effect size was 1.35, with a 95% confidence interval of 1.19-1.53, indicating a statistically significant effect (P = .001). We reported clinical tasks, baseline LLMs, retrieval sources and strategies, as well as evaluation methods. DISCUSSION: Building on our literature review, we developed Guidelines for Unified Implementation and Development of Enhanced LLM Applications with RAG in Clinical Settings to inform clinical applications using RAG. CONCLUSION: Overall, RAG implementation showed a 1.35 odds ratio increase in performance compared to baseline LLMs. Future research should focus on (1) system-level enhancement: the combination of RAG and agent, (2) knowledge-level enhancement: deep integration of knowledge into LLM, and (3) integration-level enhancement: integrating RAG systems within electronic health records." +W4399702760,,10.1186/s13643-024-02575-4,2024,Title and abstract screening for literature reviews using large language models: an exploratory study in the biomedical domain,Fabio Dennstädt,5,Systematic Reviews,151,"BACKGROUND: Systematically screening published literature to determine the relevant publications to synthesize in a review is a time-consuming and difficult task. Large language models (LLMs) are an emerging technology with promising capabilities for the automation of language-related tasks that may be useful for such a purpose. METHODS: LLMs were used as part of an automated system to evaluate the relevance of publications to a certain topic based on defined criteria and based on the title and abstract of each publication. A Python script was created to generate structured prompts consisting of text strings for instruction, title, abstract, and relevant criteria to be provided to an LLM. The relevance of a publication was evaluated by the LLM on a Likert scale (low relevance to high relevance). By specifying a threshold, different classifiers for inclusion/exclusion of publications could then be defined. The approach was used with four different openly available LLMs on ten published data sets of biomedical literature reviews and on a newly human-created data set for a hypothetical new systematic literature review. RESULTS: The performance of the classifiers varied depending on the LLM being used and on the data set analyzed. Regarding sensitivity/specificity, the classifiers yielded 94.48%/31.78% for the FlanT5 model, 97.58%/19.12% for the OpenHermes-NeuralChat model, 81.93%/75.19% for the Mixtral model and 97.58%/38.34% for the Platypus 2 model on the ten published data sets. The same classifiers yielded 100% sensitivity at a specificity of 12.58%, 4.54%, 62.47%, and 24.74% on the newly created data set. Changing the standard settings of the approach (minor adaption of instruction prompt and/or changing the range of the Likert scale from 1-5 to 1-10) had a considerable impact on the performance. CONCLUSIONS: LLMs can be used to evaluate the relevance of scientific publications to a certain review topic and classifiers based on such an approach show some promising results. To date, little is known about how well such systems would perform if used prospectively when conducting systematic literature reviews and what further implications this might have. However, it is likely that in the future researchers will increasingly use LLMs for evaluating and classifying scientific publications." +W4416403111,,10.3389/fmed.2025.1670824,2025,Clinical applications of large language models in knee osteoarthritis: a systematic review,Zebing Ma,7,Frontiers in Medicine,6,"Background and aims: Knee osteoarthritis (KOA) is a common chronic degenerative disease that significantly impacts patients' quality of life. With the rapid advancement of artificial intelligence, large language models (LLMs) have demonstrated potential in supporting medical information extraction, clinical decision-making, and patient education through their natural language processing capabilities. However, the current landscape of LLM applications in the KOA domain, along with their methodological quality, has yet to be systematically reviewed. Therefore, this systematic review aims to comprehensively summarize existing clinical studies on LLMs in KOA, evaluate their performance and methodological rigor, and identify current challenges and future research directions. Methods: Following the PRISMA guidelines, a systematic search was conducted in PubMed, Cochrane Library, Embase databases and Web of science for literature published up to June 2025. The protocol was preregistered on the OSF platform. Studies were screened using standardized inclusion and exclusion criteria. Key study characteristics and performance evaluation metrics were extracted. Methodological quality was assessed using tools such as Cochrane RoB, STROBE, STARD, and DISCERN. Additionally, the CLEAR-LLM and CliMA-10 frameworks were applied to provide complementary evaluations of quality and performance. Results: A total of 16 studies were included, covering various LLMs such as ChatGPT, Gemini, and Claude. Application scenarios encompassed text generation, imaging diagnostics, and patient education. Most studies were observational in nature, and overall methodological quality ranged from moderate to high. Based on CliMA-10 scores, LLMs exhibited upper-moderate performance in KOA-related tasks. The ChatGPT-4 series consistently outperformed other models, especially in structured output generation, interpretation of clinical terminology, and content accuracy. Key limitations included insufficient sample representativeness, inconsistent control over hallucinated content, and the lack of standardized evaluation tools. Conclusion: Large language models show notable potential in the KOA field, but their clinical application is still exploratory and limited by issues such as sample bias and methodological heterogeneity. Model performance varies across tasks, underscoring the need for improved prompt design and standardized evaluation frameworks. With real-world data and ethical oversight, LLMs may contribute more significantly to personalized KOA management. Systematic review registration: https://osf.io/jy4kz, identifier 10.17605/OSF.IO/479R8." +W7166720108,,10.1192/bjo.2026.11144,2026,Large Language Models in Dementia Assessment and Management: A Systematic Review,Ami Mehta,5,BJPsych Open,0,"Aims: Large language models (LLMs) are increasingly proposed to support healthcare delivery, yet their ability to address the distinctive challenges of dementia care remains unclear. Dementia is characterised by progressive cognitive impairment, communication barriers, high caregiver burden, and substantial demands on memory services and old age psychiatry. We systematically reviewed and classified empirical applications of LLMs in dementia assessment and care, and critically appraised the maturity of evidence, intended users, and reporting of safety, fairness, and governance relevant to clinical implementation. Methods: We searched EMBASE, MEDLINE, PsycINFO, PubMed, the ACL Anthology, the ACM Digital Library, arXiv, medRxiv, and bioRxiv (2017–June 2025). Eligible studies were empirical evaluations (simulation-based or experimental) of LLMs used for dementia-relevant clinical, caregiver-facing, or research tasks; non-evaluative commentaries were excluded. Titles/abstracts were screened by multiple reviewers, with full-text screeningconducted in accordance with PRISMA; disagreements were resolved by discussion. Using a standardised extraction form, we recorded study design, model attributes, input modalities, dementia-related use cases, evaluation setting (benchmark/vignette vs clinical data), outcomes, and reporting of ethics approval, bias assessment, and hallucination-related issues. Findings were synthesised narratively and grouped using a predefined taxonomy of clinical, patient-facing, and research-oriented applications. Results: We identified 35 dementia-focused studies, several covering multiple applications. The most frequently reported functions were screening/early detection (26/35) and diagnostic support (28/35), typically framed as classification tasks. Other common applications included research facilitation (19/35), evaluation/outcome measurement (19/35), information extraction (14/35), and prediction/risk stratification (13/35). Multimodal approaches were reported in (14/35). Intended users were most often researchers (22/35) and clinicians (20/35), with fewer studies targeting people living with dementia (10/35) or caregivers. Inputs included speech/audio (20/35) and free text (18/35); outputs most commonly included classification labels (26/35) and risk scores (11/35). Performance metrics were reported in 18/35. Reporting of safeguards was inconsistent: ethics approval (10/35, 29%), bias assessment (10/35, 29%), and explicit hallucination reporting (5/35, 14%). Conclusion: LLMs show promising experimental performance for dementia-related classification and early detection tasks, suggesting potential roles in augmenting assessment workflows. However, evidence is largely preclinical, with limited real-world validation and inconsistent reporting of fairness, safety, and governance. Future research should prioritise prospective evaluation in memory services, robust bias and hallucination assessment, and co-designed implementation with people living with dementia and caregivers to support safe, equitable integration in psychiatric practice." +W4406769232,,10.3389/fmed.2024.1512824,2025,Emerging applications of NLP and large language models in gastroenterology and hepatology: a systematic review,Mahmud Omar,6,Frontiers in Medicine,15,"Background and aim: In the last years, natural language processing (NLP) has transformed significantly with the introduction of large language models (LLM). This review updates on NLP and LLM applications and challenges in gastroenterology and hepatology. Methods: Registered with PROSPERO (CRD42024542275) and adhering to PRISMA guidelines, we searched six databases for relevant studies published from 2003 to 2024, ultimately including 57 studies. Results: Our review of 57 studies notes an increase in relevant publications in 2023-2024 compared to previous years, reflecting growing interest in newer models such as GPT-3 and GPT-4. The results demonstrate that NLP models have enhanced data extraction from electronic health records and other unstructured medical data sources. Key findings include high precision in identifying disease characteristics from unstructured reports and ongoing improvement in clinical decision-making. Risk of bias assessments using ROBINS-I, QUADAS-2, and PROBAST tools confirmed the methodological robustness of the included studies. Conclusion: NLP and LLMs can enhance diagnosis and treatment in gastroenterology and hepatology. They enable extraction of data from unstructured medical records, such as endoscopy reports and patient notes, and for enhancing clinical decision-making. Despite these advancements, integrating these tools into routine practice is still challenging. Future work should prospectively demonstrate real-world value." +W7135154108,,10.1016/j.prosdent.2026.02.009,2026,Performance of large language models conducting systematic review tasks in prosthodontics,Rata Rokhshad,5,Journal of Prosthetic Dentistry,1, +W7167597891,,10.1021/acs.jpcc.6c03290,2026,Literature Screening and Knowledge Extraction via Domain-Adaptive Transfer Learning with Large Language Models,Xitao Wang,5,The Journal of Physical Chemistry C,0,"The rapid growth of scientific literature poses major challenges for large-scale knowledge extraction in materials science, particularly when domain-specific concepts are expressed ambiguously or require deep contextual reasoning. Here, we propose a workflow that integrates domain-adaptive transfer learning, knowledge distillation, and reinforcement learning for high-precision literature screening and structured knowledge extraction in a ferroelectric Curie-temperature case study. A large open-source language model is used as a teacher to generate high-fidelity pseudolabels from full-text articles, which are then employed to train a compact student model via reinforcement-learning-based optimization. Under the same title-and-abstract input condition, the student model achieves 93.6% classification accuracy, exceeding the zero-shot Qwen3-235B-A22B baseline by 7.1 percentage points, while reducing parameter count by 138-fold and inference time by approximately 50-fold relative to the full-text teacher pipeline. This study clarifies the contextual ambiguity of “Curie temperature” in ferroelectric versus ferromagnetic literature and supports the construction of a verified ferroelectric materials knowledge base from the studied corpus. 5-fold cross-validation and ablation studies confirm the effectiveness of the proposed workflow for this task. These results demonstrate a task-specific route for combining full-text pseudolabeling with lightweight title-and-abstract screening while retaining full-text verification before database construction." +W4409830389,,10.1089/lap.2024.0277,2025,The Role of Artificial Intelligence Large Language Models in Literature Search Assistance to Evaluate Inguinal Hernia Repair Approaches,João Pedro Gonçalves Kasakewitch,8,Journal of Laparoendoscopic & Advanced Surgical Techniques,3,"Aim: This study assesses the reliability of artificial intelligence (AI) large language models (LLMs) in identifying relevant literature comparing inguinal hernia repair techniques. Material and Methods: We used LLM chatbots (Bing Chat AI, ChatGPT versions 3.5 and 4.0, and Gemini) to find comparative studies and randomized controlled trials on inguinal hernia repair techniques. The results were then compared with existing systematic reviews (SRs) and meta-analyses and checked for the authenticity of listed articles. Results: LLMs screened 22 studies from 2006 to 2023 across eight journals, while the SRs encompassed a total of 42 studies. Through thorough external validation, 63.6% of the studies (14 out of 22), including 10 identified through Chat GPT 4.0 and 6 via Bing AI (with an overlap of 2 studies between them), were confirmed to be authentic. Conversely, 36.3% (8 out of 22) were revealed as fabrications by Google Gemini (Bard), with two (25.0%) of these fabrications mistakenly linked to valid DOIs. Four (25.6%) of the 14 real studies were acknowledged in the SRs, which represents 18.1% of all LLM-generated studies. LLMs missed a total of 38 (90.5%) of the studies included in the previous SRs, while 10 real studies were found by the LLMs but were not included in the previous SRs. Between those 10 studies, 6 were reviews, and 1 was published after the SRs, leaving a total of three comparative studies missed by the reviews. Conclusions: This study reveals the mixed reliability of AI language models in scientific searches. Emphasizing a cautious application of AI in academia and the importance of continuous evaluation of AI tools in scientific investigations." +W7128197893,,10.1093/biomethods/bpag006,2026,Large language model-based multiagent collaboration for abstract screening toward automated systematic reviews,Opeoluwa Akinseloyin,3,Biology Methods and Protocols,5,"Systematic reviews (SRs) are essential for evidence-based practice but remain labor-intensive, especially during abstract screening. This study evaluates whether multiple large language models (multi-LLMs) collaboration can improve the efficiency and reduce costs for abstract screening. Abstract screening was framed as a question-answering (QA) task using cost-effective LLMs. Three multi-LLM collaboration strategies were evaluated, including majority voting by averaging opinions of peers, multi-agent debate for answer refinement, and LLM-based adjudication against answers of individual QA baselines. These strategies were evaluated on 28 SRs of the CLEF eHealth 2019 technology-assisted review benchmark using standard performance metrics such as mean average precision (MAP) and work saved over sampling at 95% recall (work saved over sampling WSS@95%). Multi-LLM collaboration significantly outperformed QA baselines. Majority voting was overall the best strategy, achieving the highest MAP 0.462 and 0.341 on subsets of SRs about clinical intervention and diagnostic technology assessment, respectively, with WSS@95% 0.606 and 0.680, enabling in theory up to 68% workload reduction at 95% recall of all relevant studies. Multi-agent debate improved weaker models most. Our own adjudicator-as-a-ranker method was the second strongest approach, surpassing adjudicator-as-a-judge, but at a significantly higher cost than majority voting and debating. Multi-LLM collaboration substantially improves abstract screening efficiency, and the success lies in model diversity. Making the best use of diversity, majority voting stands out in terms of both excellent performance and low cost compared to adjudication. Despite context-dependent gains and diminishing model diversity, multi-agent debate is still a cost-effective strategy and a potential direction of further research." +W7125913448,,10.1007/s44163-025-00784-x,2026,Large language models for clinical artificial intelligence in healthcare a systematic review,Rawan Ghnemat,2,Discover Artificial Intelligence,9,"Abstract Large Language Models (LLMs) have demonstrated the capacity to process, reason, and generate extensive volumes of data, providing a novel paradigm for integrating generative artificial intelligence (GenAI) into the medical field. Multimodal LLMs (MLLMs) extend these capabilities by incorporating diverse data modalities into unified representations, including genomics, medical imaging, and clinical text. This systematic review synthesizes advancements from 246 records identified between January 2020 and September 2025, of which 90 studies were included after full-text screening, to address critical gaps in understanding the clinical role of LLM and MLLM in healthcare. We trace the evolution from classical natural language processing (NLP) approaches to modern transformer-based architectures, summarize their technical foundations, and examine their construction, evaluation, and deployment in medical workflows. Key contributions include highlighting multimodal integration (e.g., imaging-genomics-text fusion), ethical governance frameworks, and validated domain-specific fine-tuning in clinical settings. We also highlight advances in Prompting, Retrieval-Augmented Generation (RAG), and Multi-Agent (agentic) workflows, providing a critical assessment of their benefits and limitations. In addition, we analyze challenges such as hallucinations, bias, and privacy risks, while providing actionable guidelines for clinicians, developers, and policymakers to improve regulatory compliance. By consolidating the nomenclature and systematically evaluating GenAI in medicine, this review offers evidence-based recommendations and directions for the safe and effective integration of generative AI into healthcare. The findings are intended as an authoritative guide for researchers and practitioners, bridge principles, clinical applications, and policy considerations for LLM and MLLM." +W4415976151,10.48550/arxiv.2504.02917,10.48550/arxiv.2504.02917,2025,Bias in Large Language Models Across Clinical Applications: A Systematic Review,Thanathip Suenghataiphorn,4,arXiv (Cornell University),8,"Background: Large language models (LLMs) are rapidly being integrated into healthcare, promising to enhance various clinical tasks. However, concerns exist regarding their potential for bias, which could compromise patient care and exacerbate health inequities. This systematic review investigates the prevalence, sources, manifestations, and clinical implications of bias in LLMs. Methods: We conducted a systematic search of PubMed, OVID, and EMBASE from database inception through 2025, for studies evaluating bias in LLMs applied to clinical tasks. We extracted data on LLM type, bias source, bias manifestation, affected attributes, clinical task, evaluation methods, and outcomes. Risk of bias was assessed using a modified ROBINS-I tool. Results: Thirty-eight studies met inclusion criteria, revealing pervasive bias across various LLMs and clinical applications. Both data-related bias (from biased training data) and model-related bias (from model training) were significant contributors. Biases manifested as: allocative harm (e.g., differential treatment recommendations); representational harm (e.g., stereotypical associations, biased image generation); and performance disparities (e.g., variable output quality). These biases affected multiple attributes, most frequently race/ethnicity and gender, but also age, disability, and language. Conclusions: Bias in clinical LLMs is a pervasive and systemic issue, with a potential to lead to misdiagnosis and inappropriate treatment, particularly for marginalized patient populations. Rigorous evaluation of the model is crucial. Furthermore, the development and implementation of effective mitigation strategies, coupled with continuous monitoring in real-world clinical settings, are essential to ensure the safe, equitable, and trustworthy deployment of LLMs in healthcare." +W4414983308,,10.1136/bmjdhai-2025-000017,2025,Large language model enhanced framework for systematic reviews and meta-analyses,Jiashu Shen,6,BMJ Digital Health & AI,2,"Objective: To evaluate and synthesise current applications of large language models (LLMs) in systematic reviews and meta-analyses (SRMAs), identify key limitations and propose an enhanced theoretical framework to improve the efficiency, scalability and reliability of evidence synthesis. Methods and analysis: We conducted a narrative review of recent studies applying LLMs across key SRMA stages. A total of 21 publications were analysed for model type, task application, accuracy metrics and workflow impact. Building on this evidence base, we designed a comprehensive LLM-enhanced SRMA framework that categorises LLM roles as consultants and assistants, integrates human-in-the-loop strategies and uses retrieval-augmented generation (RAG) and agent-based architectures to address critical challenges including hallucinations, bias and workflow inefficiency. Results: The reviewed literature demonstrated that LLMs can support various SRMA tasks with reported accuracy ranging from 61% to 99%, showing particular promise in literature screening and data extraction. Our proposed framework conceptualises modular integration of LLMs across all six SRMA stages, with LLMs serving as consultants for research question formulation and search strategy development and as assistants for task automation including abstract screening and structured data extraction. The framework incorporates RAG technology to reduce hallucinations by grounding outputs in retrieved literature and employs agent-based orchestration for complex analytical workflows. Theoretical analysis suggests potential for significant efficiency gains while maintaining methodological rigour through strategic human oversight. Conclusion: LLMs offer substantial theoretical potential to transform evidence synthesis by improving efficiency, scalability and consistency across SRMA workflows. The proposed LLM-enhanced framework provides a systematic, theoretically grounded approach for integrating advanced artificial intelligence capabilities into existing SRMA methodologies while preserving essential human oversight and analytical integrity. Future empirical studies are needed to validate the framework's practical effectiveness, establish implementation protocols and demonstrate real-world benefits in evidence-based medicine." +W4408987202,,10.1101/2025.03.28.25324834,2025,Prompt Engineering in Large Language Models for Patient Education: A Systematic Review,Aya Mudrik,5,medRxiv,7,"ABSTRACT Background Large language models (LLMs) have shown promise in generating patient-friendly medical content, but their outputs often vary in accuracy, readability, and relevance. Prompt engineering—structuring inputs to guide LLM responses—may improve the quality of educational materials, yet its impact on patient education remains unclear. Objectives To systematically review whether prompt engineering improves readability, accuracy, and usability of LLM-generated content for patient education. Methods We conducted a systematic review in accordance with PRISMA guidelines. PubMed, Scopus, and Web of Science were searched for original studies evaluating prompt engineering techniques in patient education. Data were extracted on prompt types, LLM models used, and outcomes. Risk of bias was assessed using the QUADAS-2 tool, and a narrative synthesis was performed. Results Our search identified five studies that met our criteria, focusing on answering patient questions and generating medical information. Prompt engineering techniques included instruction-based, elaborated, role-defining, scene-defining, and domain-specific prompts. Structured prompting improved accuracy and comprehensiveness in several cases, particularly when specific formats or custom instructions were used. Readability gains were notable when prompts explicitly requested simpler language and reading levels, though some strategies unintentionally increased complexity. Variability in effectiveness across LLMs and prompt types was observed. Conclusion Prompt engineering can enhance the clarity and, in some cases, the accuracy of LLM-generated patient education materials. However, benefits vary by model and strategy. Standardized approaches and further research are needed to optimize prompts, minimize bias, and support reliable, accessible patient communication." +W7114905629,,10.1093/bib/bbaf631.059,2025,Should we use large language models to perform systematic reviews?,TH Chan,4,Briefings in Bioinformatics,0,"Abstract Background Large language models (LLMs) have shown promising performances in literature search, screening of full-text scientific articles, data extraction, and produce a written summary. LLMs have a strong potential in automating the process of conducting systematic reviews. Nonetheless, conducting systematic reviews is also a training exercise for critical appraisal of scientific evidence and synthesis of scientific claims. Overreliance on LLMs may erode comprehension and analytical skills in reading scientific articles over time, leading to cognitive offloading. Other risks may involve hallucination and responsibility problem. Methods To address both opportunities and limitations of LLMs when applied in systematic reviews, the LLM-Augmented Systematic Evidence Review (LASER) framework is proposed. Results The framework recommends different degree of LLM usage in different steps in the systematic review process to promote efficiency, ensure accuracy, and prevent cognitive offloading of the researcher involved. In this framework, the specific degree of LLM usage may depend on the career stage of the researcher and the accuracy of the LLM." +W4416347706,,10.3352/jeehp.2025.22.36,2025,Performance of large language models in medical licensing examinations: a systematic review and meta-analysis,Haniyeh Nouri,6,Journal of Educational Evaluation for Health Professions,14,"PURPOSE: This study systematically evaluates and compares the performance of large language models (LLMs) in answering medical licensing examination questions. By conducting subgroup analyses based on language, question format, and model type, this meta-analysis aims to provide a comprehensive overview of LLM capabilities in medical education and clinical decision-making. METHODS: This systematic review, registered in PROSPERO and following PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) guidelines, searched MEDLINE (PubMed), Scopus, and Web of Science for relevant articles published up to February 1, 2025. The search strategy included Medical Subject Headings (MeSH) terms and keywords related to (""ChatGPT"" OR ""GPT"" OR ""LLM variants"") AND (""medical licensing exam*"" OR ""medical exam*"" OR ""medical education"" OR ""radiology exam*""). Eligible studies evaluated LLM accuracy on medical licensing examination questions. Pooled accuracy was estimated using a random-effects model, with subgroup analyses by LLM type, language, and question format. Publication bias was assessed using Egger's regression test. RESULTS: This systematic review identified 2,404 studies. After removing duplicates and excluding irrelevant articles through title and abstract screening, 36 studies were included after full-text review. The pooled accuracy was 72% (95% confidence interval, 70.0% to 75.0%) with high heterogeneity (I2=99%, P<0.001). Among LLMs, GPT-4 achieved the highest accuracy (81%), followed by Bing (79%), Claude (74%), Gemini/Bard (70%), and GPT-3.5 (60%) (P=0.001). Performance differences across languages (range, 62% in Polish to 77% in German) were not statistically significant (P=0.170). CONCLUSION: LLMs, particularly GPT-4, can match or exceed medical students' examination performance and may serve as supportive educational tools. However, due to variability and the risk of errors, they should be used cautiously as complements rather than replacements for traditional learning methods." +W4394009806,,10.1016/j.ajic.2024.03.016,2024,Utilizing natural language processing and large language models in the diagnosis and prediction of infectious diseases: A systematic review,Mahmud Omar,4,American Journal of Infection Control,59, +W7222962256,,10.83095/ceurws_04296_paper1,2026,Fine-Tuning A 1.2B Large Language Model for Systematic Review Screening in Educational Technology,Kweku Andoh Yamoah,5,ceur-ws.org,0, +W4413467640,,10.2196/76252,2025,Automated Literature Screening for Hepatocellular Carcinoma Treatment Through Integration of 3 Large Language Models: Methodological Study,Chen Pan,6,JMIR Medical Informatics,6,"BACKGROUND Primary liver cancer, particularly hepatocellular carcinoma (HCC), poses significant clinical challenges due to late-stage diagnosis, tumor heterogeneity, and rapidly evolving therapeutic strategies. While systematic reviews and meta-analyses are essential for updating clinical guidelines, their labor-intensive nature limits timely evidence synthesis. OBJECTIVE This study proposes an automated literature screening workflow powered by large language models (LLMs) to accelerate evidence synthesis for HCC treatment guidelines. METHODS We developed a tripartite LLM framework integrating Doubao-1.5-pro-32k, Deepseek-v3, and DeepSeek-R1-Distill-Qwen-7B to simulate collaborative decision-making for study inclusion and exclusion. The system was evaluated across 9 reconstructed datasets derived from published HCC meta-analyses, with performance assessed using accuracy, agreement metrics (κ and prevalence-adjusted bias-adjusted κ), recall, precision, F1-scores, and computational efficiency parameters (processing time and cost). RESULTS The framework demonstrated good performance, with a weighted accuracy of 0.96 and substantial agreement (prevalence-adjusted bias-adjusted κ=0.91), achieving high weighted recall (0.90) but modest weighted precision (0.15) and F1-scores (0.22). Computational efficiency varied across datasets (processing time: 248-5850 s; cost: US $0.14-$3.68 per dataset). CONCLUSIONS This LLM-driven approach shows promise for accelerating evidence synthesis in HCC care by reducing screening time while maintaining methodological rigor. Key limitations related to clinical context sensitivity and error propagation highlight the need for reinforcement learning integration and domain-specific fine-tuning. LLM agent architectures with reinforcement learning offer a practical path for streamlining guideline updates, though further optimization is needed to improve specialization and reliability in complex clinical settings." +W4386399156,,10.1101/2023.09.03.23294842,2023,How Large Language Models Perform on the United States Medical Licensing Examination: A Systematic Review,Dana Brin,6,medRxiv,16,"ABSTRACT Objective The United States Medical Licensing Examination (USMLE) assesses physicians’ competency and passing is a requirement to practice medicine in the U.S. With the emergence of large language models (LLMs) like ChatGPT and GPT-4, understanding their performance on these exams illuminates their potential in medical education and healthcare. Materials and Methods A literature search following the 2020 PRISMA guidelines was conducted, focusing on studies using official USMLE questions and publicly available LLMs. Results Three relevant studies were found, with GPT-4 showcasing the highest accuracy rates of 80-90% on the USMLE. Open-ended prompts typically outperformed multiple-choice ones, with 5-shot prompting slightly edging out zero-shot. Conclusion LLMs, especially GPT-4, display proficiency in tackling USMLE-standard questions. While the USMLE is a structured evaluation tool, it may not fully capture the expansive capabilities and limitations of LLMs in medical scenarios. As AI integrates further into healthcare, ongoing assessments against trusted benchmarks are essential." +W7223446955,,10.83095/ceurws_04296_001,2026,Fine-Tuning A 1.2B Large Language Model for Systematic Review Screening in Educational Technology,Kweku Andoh Yamoah,5,ceur-ws.org,0, +W7172325417,,10.1017/s0266462326104073,2026,A prompt-based tutorial for large language model–assisted screening in systematic reviews and meta-analyses,Tongyu Qiu,4,International Journal of Technology Assessment in Health Care,0,"OBJECTIVES: Title/abstract and full-text screening are among the most time-consuming stages of systematic reviews. Large language models (LLMs) such as ChatGPT can assist in screening, but prior evaluations have largely focused on abstract-level decisions and require coding or application programming interfaces (API) access, limiting practical application. This study developed and evaluated a structured, prompt-based approach that enables LLMs to perform both abstract and full-text screening without programming or API integration. METHODS: Using datasets from three completed meta-analyses, we implemented a stepwise training framework involving comprehension checks, criterion-specific feedback, and iterative prompt refinement. Study 1 optimized file format and batch size using 1,000 abstracts and 50 full texts. Studies 2 and 3 validated the approach using independent datasets on teletherapy versus in-person therapy (1,321 abstracts, 82 full texts) and emotion regulation and mental health (1,500 abstracts, 129 full texts), respectively. Human reviewers' decisions served as the reference standard. RESULTS: In Study 1, the LLM achieved 98 percent sensitivity and 80.6 percent specificity at abstract screening, with batches of 50 abstracts in plain-text format performing optimally. In Study 2, sensitivity and specificity were 100 percent and 85.6 percent for abstracts, with 82.1 percent full-text accuracy and all eligible studies retained. In Study 3, abstract screening achieved 100 percent sensitivity and 93 percent specificity, while full-text screening achieved 96.2 percent sensitivity and 90.3 percent specificity. CONCLUSIONS: A structured, prompt-based approach enabled LLMs to achieve high sensitivity and specificity across both screening stages without coding or APIs. With human oversight, this accessible workflow can reduce reviewer burden and accelerate evidence synthesis." +W4414532163,,10.1007/s10815-025-03679-z,2025,The potential of large language models in the field of infertility: a systematic review,Wei Li,11,Journal of Assisted Reproduction and Genetics,1, +W7152014256,,10.1371/journal.pdig.0001354,2026,A systematic review of the limitations of large language models in generating healthcare content,Mohsen Khosravi,5,PLOS Digital Health,5,"Large language models (LLMs) have recently gained prominence in healthcare content provision due to their numerous advantages. Despite these benefits, LLMs exhibit notable limitations in this domain. This study aimed to systematically identify the limitations of LLMs in provision of healthcare content. This study was a systematic review conducted in September 2025, including articles published in English between 2018 and 2025. Searches were performed in PubMed, Scopus, and the Cochrane Database of Systematic Reviews. Two independent evaluators screened the references and assessed quality of the selected studies using the Authority, Accuracy, Coverage, Objectivity, Date, and Significance (AACODS) checklist. Data were analyzed using Boyatzis's qualitative thematic approach with an inductive methodology, applying the input-process-output (IPO) model as the analytical framework. A total of 81 studies were included in the final analysis. The included studies were predominantly of high quality and demonstrated minimal risk of bias. The thematic analysis identified key themes: data limitations, dependence on input and prompt quality, accessibility issues, model design and architecture constraints, interaction challenges, response quality and comprehensiveness, and ethical, safety, and regulatory concerns. The study identified multiple limitations of LLMs in healthcare, with output issues being most common. In this regard, the most frequently cited limitation was the accuracy gap. However, these output issues were mainly resulted from flaws in input data, emphasizing the crucial role of input quality. The study also proposed strategies to address these challenges." +W4410421218,,10.1016/j.identj.2025.04.015,2025,Can Large Language Models Serve as Reliable Tools for Information in Dentistry? A Systematic Review,Nora Alhazmi,6,International Dental Journal,24,"Large language models (LLMs) have gained popularity among dental students for generating subject-related answers. However, their widespread use raises significant concerns about misinformation. This systematic review aims to critically evaluate studies assessing the performance of LLMs in dentistry. A comprehensive electronic search was conducted in PubMed/Medline, Scopus, Embase, Web of Science, Google Scholar, and the Saudi Digital Library to identify studies published up to September 2024. The study quality was assessed using the Prediction Model Risk of Bias Assessment Tool (PROBAST). A total of 2030 studies have been identified. After removing 907 duplicate records, 1123 studies remained for screening. Ultimately, 31 studies met the inclusion criteria. Approximately half of these studies were classified as ""high risk,"" while the remainder were classified as ""low risk."" The applicability of the findings was rated as ""low concern."" The primary limitations of LLMs include their inability to specify information sources and their tendency to generate fabricated citations. Based on this review, LLMs hold promise as supplementary educational tools in dentistry. Evidence indicates that students using LLMs may achieve improved academic performance compared to traditional methods. However, concerns about occasional inaccuracies and unreliable citations underscore the need for further research, integration with validated sources, and adherence to ethical guidelines. Ultimately, LLMs should be viewed as complementary tools within dental education, with careful consideration of their limitations." +W4414532985,,10.1200/cci-25-00076,2025,Mitigating Ethical Issues for Large Language Models in Oncology: A Systematic Review,Shuang Zhou,9,JCO Clinical Cancer Informatics,4,"PURPOSE: Large language models (LLMs) have demonstrated remarkable versatility in oncology applications, such as cancer staging and survival analysis. Despite their potential, ethical concerns such as data privacy breaches, bias in training data, lack of transparency, and risks associated with erroneous outputs pose significant challenges to their adoption in high-stakes oncology settings. Therefore, we aim to explore the ethical challenges associated with LLM-based applications in oncology and evaluate emerging techniques designed to address these issues. METHODS: Following the Preferred Reporting Items for Systematic Reviews and Meta-Analyses framework, a systematic review was conducted to evaluate publications related to ethical issues of LLMs in oncology across eight academic databases (eg, PubMed, Web of Science, and Embase) between January 1, 2019, and December 31, 2024. RESULTS: The search retrieved 4,319 published articles, of which 65 publications were preserved and included in our analysis. We identified six prevalent ethical challenges in oncology, including trust, equity, privacy, transparency, nonmaleficence, and accountability. We then evaluated emerging technical solutions to mitigate ethical challenges and summarized evaluation metrics used to assess these solutions' effectiveness. CONCLUSION: This review provides actionable recommendations for responsibly deploying LLMs in oncology, ensuring adherence to ethical guidelines, and fostering improved patient outcomes. By bridging technical and clinical perspectives, this review offers a foundational framework for advancing ethical artificial intelligence applications in oncology and highlights areas for future research." +W4411407939,,10.34133/hds.0322,2025,LitAutoScreener: Development and Validation of an Automated Literature Screening Tool in Evidence-Based Medicine Driven by Large Language Models,Yiming Tao,6,Health Data Science,9,"Background: The traditional manual literature screening approach is limited by its time-consuming nature and high labor costs. A pressing issue is how to leverage large language models to enhance the efficiency and quality of evidence-based evaluations of drug efficacy and safety. Methods: This study utilized a manually curated reference literature database—comprising vaccine, hypoglycemic agent, and antidepressant evaluation studies—previously developed by our team through conventional systematic review methods. This validated database served as the gold standard for the development and optimization of LitAutoScreener. Following the PICOS (Population, Intervention, Comparison, Outcomes, Study Design) principles, a chain-of-thought reasoning approach with few-shot learning prompts was implemented to develop the screening algorithm. We subsequently evaluated the performance of LitAutoScreener using 2 independent validation cohorts, assessing both classification accuracy and processing efficiency. Results: For respiratory syncytial virus vaccine safety validation title–abstract screening, our tools based on GPT (GPT-4o), Kimi (moonshot-v1-128k), and DeepSeek (deepseek-chat 2.5) demonstrated high accuracy in inclusion/exclusion decisions (99.38%, 98.94%, and 98.85%, respectively). Recall rates were 100.00%, 99.13%, and 98.26%, with statistically significant performance differences ( χ 2 = 5.99, P = 0.048), where GPT outperformed the other models. Exclusion reason concordance rates were 98.85%, 94.79%, and 96.47% ( χ 2 = 30.22, P < 0.001). In full-text screening, all models maintained perfect recall (100.00%), with accuracies of 100.00% (GPT), 100.00% (Kimi), and 99.45% (DeepSeek). Processing times averaged 1 to 5 s per article for title–abstract screening and 60 s for full-text processing (including PDF preprocessing). Conclusions: LitAutoScreener offers a new approach for efficient literature screening in drug intervention studies, achieving high accuracy and significantly improving screening efficiency." +W7151398448,,10.47626/2237-6089-2025-1237,2026,The Role of Large Language Models in Teaching Psychiatric Semiology: A Systematic Review,Vinícius Vicente Soares,5,Trends in Psychiatry and Psychotherapy,0,"INTRODUCTION: Teaching psychiatric semiology faces challenges such as the limited availability of real patients for educational purposes and a shortage of specialized instructors. This systematic review investigated the applications of Large Language Models (LLMs), exemplified by ChatGPT, to address these gaps. METHODS: Studies published between November 2022 and September 17, 2025, were identified through searches in PubMed/MEDLINE, Scopus, Web of Science, IEEE Xplore, EBSCO, BVS, and EMBASE databases. Two independent reviewers conducted the systematic review following the PRISMA 2020 checklist, and the methodological quality of the included studies was individually assessed using the Mixed Methods Appraisal Tool (MMAT). RESULTS: Of the 1,549 studies initially identified, two met the inclusion criteria, involving medical students and educators in Canada and Switzerland. LLMs demonstrated performance comparable to human experts in creating diagnostic tests and clinical vignettes but exhibited occasional simplifications and algorithmic biases. Participants reported positive perceptions of the tools' efficiency and practicality, but emphasized the need for specialized supervision and attention to potential privacy issues and superficiality in complex cases. CONCLUSIONS: LLMs show potential as valuable supplementary resources for teaching psychiatric semiology, especially in settings with shortages of teachers and available patients for training. However, their limitations, including cultural and algorithmic biases, potential privacy risks, and superficial content generation, require close monitoring by experienced professionals. Although current evidence is preliminary, prospects are promising, highlighting the need for more robust and long-term studies to evaluate the educational use of LLMs." +W4320342216,10.48550/arxiv.2302.03495,10.48550/arxiv.2302.03495,2023,Can ChatGPT Write a Good Boolean Query for Systematic Review Literature Search?,Shuai Wang,4,arXiv (Cornell University),40,"Systematic reviews are comprehensive reviews of the literature for a highly focused research question. These reviews are often treated as the highest form of evidence in evidence-based medicine, and are the key strategy to answer research questions in the medical field. To create a high-quality systematic review, complex Boolean queries are often constructed to retrieve studies for the review topic. However, it often takes a long time for systematic review researchers to construct a high quality systematic review Boolean query, and often the resulting queries are far from effective. Poor queries may lead to biased or invalid reviews, because they missed to retrieve key evidence, or to extensive increase in review costs, because they retrieved too many irrelevant studies. Recent advances in Transformer-based generative models have shown great potential to effectively follow instructions from users and generate answers based on the instructions being made. In this paper, we investigate the effectiveness of the latest of such models, ChatGPT, in generating effective Boolean queries for systematic review literature search. Through a number of extensive experiments on standard test collections for the task, we find that ChatGPT is capable of generating queries that lead to high search precision, although trading-off this for recall. Overall, our study demonstrates the potential of ChatGPT in generating effective Boolean queries for systematic review literature search. The ability of ChatGPT to follow complex instructions and generate queries with high precision makes it a valuable tool for researchers conducting systematic reviews, particularly for rapid reviews where time is a constraint and often trading-off higher precision for lower recall is acceptable." +W7143806077,,10.3389/fmed.2026.1810098,2026,Large language models for primary care ophthalmic education: a systematic review,Shuang Li,7,Frontiers in Medicine,2,"Background: Primary care physicians (PCPs) are a critical first contact for eye-health screening, risk stratification, and referral, yet ophthalmology training and point-of-care support in primary care remain insufficient. Recent advances in generative artificial intelligence (generative AI), particularly large language models (LLMs), may help address these gaps through conversational, scenario-based learning and structured feedback. However, the educational effectiveness, reproducibility, and safety boundaries of LLM-enabled tools in primary care ophthalmology remain unclear. Methods: We conducted a systematic review of studies evaluating or applying LLMs in ophthalmic education, training, assessment, or primary care-relevant clinical support. PubMed, Web of Science Core Collection, and Scopus were searched from January 1, 2020 to December 31, 2025 using combined terms related to LLMs/generative AI, ophthalmology, and education or assessment. Citation chaining was also performed to reduce omission. Two reviewers independently screened records and extracted data. Results: The evidence base is dominated by vignette-based benchmarks, comparative scoring studies, and evaluations conducted in limited-sample or controlled settings; prospective real-world validation using learner transfer, clinical behavior change, workflow impact, or patient outcomes remains scarce. Across studies, LLMs can serve as ""cognitive apprenticeship"" partners by externalizing clinical reasoning and enabling repeated practice in key-feature extraction, differential diagnosis, risk stratification, and referral-threshold decisions. Applications include triage/reasoning drills, virtual patient interviewing, and support for structured referrals, documentation, and patient education, often strengthened by retrieval-augmented generation. Most studies benchmarked outputs against expert consensus or guidelines, but scoring rubrics and reference standards varied widely, limiting cross-study comparability. Some reports noted that adding clinical photographs could reduce accuracy, suggesting current multimodal models are better suited for history-based reasoning than fine-grained image interpretation. Limitations include heterogeneity, rapid model iteration, reproducibility challenges, multimodal instability, and safety risks such as hallucination, bias, and automation bias. Commonly recommended safeguards include retrieval grounding, source attribution, red-flag checklists, and human-in-the-loop review. Conclusion: With clearly defined task scopes and robust safeguards, LLMs may improve the accessibility and efficiency of primary care ophthalmic education, but should augment rather than replace expert judgment. Future work should prioritize pragmatic multicenter trials, mixed-method implementation studies, and standardized cross-lingual evaluations to define safe and effective implementation pathways." +W7129095804,,10.17605/osf.io/rzju8,2026,Performance of Large Language Models in Title and Abstract Screening of Preclinical Systematic Reviews - Study Registration,Francisco Marques da Silva Neto,4,,0,"This preregistered methodological validation study evaluates how well large language models perform title/abstract screening for preclinical systematic reviews from BRISA. We will compare LLM decisions against the reviews’ final full-text inclusion (gold standard), estimating sensitivity/specificity, efficiency gains, and test–retest reproducibility across multiple datasets." +W7169782079,,10.1108/jstpm-01-2026-0015,2026,Hallucinations in large language models within academic contexts: a systematic review,Ke En Lai,4,Journal of Science and Technology Policy Management,1,"Purpose This study aims to examine the phenomenon of hallucinations in large language models (LLMs) within academic contexts, focusing on their manifestations, causes and implications for academic integrity, research quality and responsible artificial intelligence adoption in higher education. Design/methodology/approach A systematic literature review was conducted in accordance with PRISMA 2020 guidelines. Searches across Scopus, Web of Science and Emerald Insight databases using keywords related to AI hallucination and academic applications, of which 25 peer-reviewed journal articles met the inclusion criteria. Qualitative thematic analysis was performed using NVivo 14 to synthesise evidence on hallucination types, academic applications, impacts and mitigation strategies. Findings Six recurring types of hallucinations were identified, with fabricated or inaccurate citations emerging as the most prevalent. The findings indicate that hallucinations systematically compromise academic writing quality, distort assessment processes and undermine epistemic trust in scholarly outputs. Variation in hallucination rates across models and disciplines highlights their context-dependent nature. Key contributing factors include probabilistic text generation, limitations in training data, insufficient contextual understanding and the absence of robust verification mechanisms. Practical implications It further contributes a structured classification of hallucination types and a multi-layered governance approach to inform institutional policy and responsible AI adoption. Social implications Addressing hallucinations in academic knowledge production is essential for preserving public trust in higher education and safeguarding the societal value of scholarly research. Originality/value This study advances existing knowledge by developing an integrated conceptual perspective linking hallucinations to epistemic risk, information integrity and digital trust." +W7127186092,,10.1007/s10586-025-05891-z,2026,"The rise of hallucination in large language models: systematic reviews, performance analysis and challenges",Shamsu Abdullahi,3,Cluster Computing,5, +W7155029538,,10.1145/3842759,2026,LLM4Log: A Systematic Review of Large Language Model-based Log Analysis,Zeyang Ma,3,ACM Transactions on Software Engineering and Methodology,1,"Software systems generate massive, evolving, semi-structured logs that are central to reliability engineering and AIOps, yet difficult to analyze at scale under drift and limited labels. Recent advances in pretrained Transformer models and instructiontuned large language models (LLMs) have reshaped log analysis by enabling semantic generalization and cross-source evidence integration, but also introducing deployment risks such as context limits, latency and cost, privacy constraints, and hallucinations. This paper presents LLM4Log , a systematic review of LLM-based log analysis across the end-to-end pipeline, from upstream logging-statement generation and maintenance to log parsing/structuring and downstream tasks including anomaly detection, failure prediction, root cause analysis, and log summarization. Following a structured search and manual screening protocol, we completed literature collection in November 2025 and identified 145 unique papers across seven logging tasks. We organize the research area through a unified, task-driven taxonomy, summarize common design patterns (prompting/ICL, retrieval grounding, fine-tuning, tool/agent augmentation, and verification), and analyze evaluation practices, datasets, metrics, and reproducibility. Based on these cross-paper analyses, we summarize key lessons and open challenges for reliable real-world adoption. We emphasize robustness under drift and long-tail events, grounding and faithfulness for operator-facing outputs, and deployment-oriented designs with verifiable behavior." +W7113238458,,,2025,The Social Science of Large Language Models: A Systematic Review and Computational Taxonomy,Xiao Jia,1,OSF Preprints (OSF Preprints),0,"This project preregisters a systematic review protocol on the social science of large language models (LLMs). LLMs such as GPT, Claude, and LLaMA are increasingly used in contexts that intersect with human social behavior, yet existing studies are scattered across disciplines. We will systematically search Web of Science, Scopus, ACM Digital Library, IEEE Xplore, PubMed, arXiv, Consensus, and Semantic Scholar, supplemented with hand-searching of top journals and conferences. Inclusion criteria focus on studies that directly involve LLMs and social science constructs; exclusion criteria filter out purely engineering applications and studies where LLMs are only used as tools. Screening will follow PRISMA 2020 guidelines, with two reviewers conducting independent title/abstract and full-text reviews. The expected outcome is a comprehensive overview of ~150 core studies, organized into a conceptual framework that highlights LLMs’ capacities and limitations. To ensure systematic categorization and to handle the large body of literature, we will employ an unsupervised machine learning approach to generate a computational taxonomy of the studies. This will involve clustering studies based on content similarity using sentence embeddings and dimensionality reduction techniques like UMAP. The resulting taxonomy will reveal the latent structure of the field and identify key research themes across various social science domains. This review will serve as a resource for researchers in AI, psychology, sociology, political science, and economics, and will provide methodological guidance for future work. Data extraction templates and coding sheets will be shared openly to enhance transparency and reproducibility." +W4416947049,,10.2196/84120,2025,Knowledge-Practice Performance Gap in Clinical Large Language Models: Systematic Review of 39 Benchmarks,Eun Jeong Gong,4,Journal of Medical Internet Research,41,"BACKGROUND: The evaluation of large language models (LLMs) in medicine has undergone a shift from knowledge-based testing to practice-based assessment, representing an evolution in how we measure artificial intelligence readiness for clinical deployment. While LLMs now routinely exceed human performance on medical licensing examinations, their translation to clinical practice remains poorly characterized. OBJECTIVE: This systematic review aims to categorize and analyze medical LLM benchmarks, examining performance patterns across different evaluation paradigms and identifying gaps in current assessment methodologies. METHODS: The protocol was registered at PROSPERO (CRD420251139729). Four databases (MEDLINE/PubMed, Embase/Ovid, Cochrane Library, and arXiv) were searched from inception to August 31, 2025, using keywords related to clinical medicine benchmarks in LLMs. Studies were included if they (1) investigated clinical medicine benchmarks in LLMs, (2) were published in English, and (3) were available in full-text. Studies were excluded if they evaluated nonmedical domains or lacked benchmark validation. Methodological quality was assessed using the Mixed Methods Appraisal Tool (version 2018) by 2 independent reviewers (κ=0.91). Due to heterogeneity in evaluation metrics preventing meta-analysis, narrative synthesis was conducted using structured categorization of benchmark types. RESULTS: From 3917 screened records, 39 medical LLM benchmarks were identified and categorized into 21 (54%) knowledge-based, 15 (38%) practice-based, and 3 (8%) hybrid frameworks. These benchmarks collectively encompass over 2.3 million questions across 45 languages and 172 medical specialties. Traditional knowledge-based benchmarks show saturation with leading models achieving 84%-90% accuracy on USMLE (United States Medical Licensing Examination)-style examinations, approaching or exceeding average physician performance. However, practice-based assessments reveal performance challenges, with specific benchmarks showing varied results: DiagnosisArena 45.82% (95% CI 42.9%-48.8%), MedAgentBench 69.67% (95% CI 64.2%-74.6%), and HealthBench 60% (95% CI 58.6%-61.3%) success rates, with practice-based benchmarks showing lower performance (45%-69%) compared to knowledge benchmarks (84%-90%). Task-specific analysis revealed differential performance patterns: factual retrieval maintained 85%-93% accuracy, clinical reasoning dropped to 50%-60%, diagnostic tasks achieved 45%-55% success, and safety assessment showed significant gaps at 40%-50% accuracy despite being life-critical. Geographic representation spans 6 continents with 18 (46%) benchmarks, incorporating non-English content. Quality assessment revealed 26% (10/39) of benchmarks had insufficient methodological reporting for complete evaluation. CONCLUSIONS: This systematic review provides the first comprehensive analysis quantifying the significant ""knowledge-practice gap"" in medical artificial intelligence: high performance on knowledge-based examinations (84%-90%) does not translate to clinical competence (45%-69%), with safety assessments at 40%-50%. Our findings provide quantitative evidence for regulators and health systems that examination scores are insufficient and misleading proxies for clinical readiness. This review concludes that autonomous deployment is not currently justifiable and that all evidence-based implementation strategies must mandate practice-oriented validation and robust human-in-the-loop oversight to ensure patient safety. TRIAL REGISTRATION: PROSPERO CRD420251139729; https://www.crd.york.ac.uk/PROSPERO/view/CRD420251139729." +W7223172290,,10.1038/s41746-026-03349-z,2026,Large language models for multimodal clinical predictive modeling: a systematic review,Levi Kaster,4,npj Digital Medicine,0,"The electronic health record (EHR) comprises structured and unstructured data, both of which are integral for comprehensively representing patient history. The emergence of large language models (LLMs) has enabled the incorporation of unstructured data modalities into machine learning models that leverage EHR data. In this systematic review, we characterize studies that utilize LLM-enabled multimodal predictive modeling with EHR data. Following a screening of 641 studies, 57 met inclusion criteria. We identified three strategies for LLM-based multimodal integration: embedding, extraction, and text fusion. Delineated by the format of the unstructured data at the time of fusion, all strategies yielded performance gains (average 6.4% increase) over single modality models. LLMs facilitated modeling by generating note-embeddings for downstream prediction, extracting clinically informative variables, and summarizing long text documents prior to integration. Building on our review, we identified LLM-enhanced predictive modeling considerations that include sample size, interpretability, and computational costs. This study supports the potential value of LLMs for EHR-based predictive modeling and provides a framework for future multimodal modeling tools." +W4411253453,,10.1101/2025.06.10.25329350,2025,Large Language Models for Sentiment Analysis in Healthcare: A Systematic Review Protocol,Ravi Shankar,3,medRxiv,0,"Abstract Large language models (LLMs) have emerged as powerful tools for sentiment analysis in healthcare, offering potential advantages in capturing contextual information and semantic relationships in complex medical text. Healthcare sentiment analysis presents unique challenges due to domain-specific terminology, privacy regulations, and the nuanced nature of patient experiences. This systematic review protocol outlines a comprehensive methodology to investigate the application of LLMs for sentiment analysis across healthcare settings, including analysis of patient feedback, social media content, and electronic health records. By synthesizing current evidence, we aim to provide insights for researchers, clinicians, and policymakers on the effectiveness, limitations, and ethical considerations of these advanced natural language processing techniques. We will conduct a systematic review following PRISMA-P 2015 guidelines and using the PICOS framework. The search strategy will encompass eight major databases (PubMed, Web of Science, Embase, CINAHL, MEDLINE, The Cochrane Library, PsycINFO, and Scopus) using a comprehensive search string combining terms related to LLMs, sentiment analysis, and healthcare contexts. We will include peer-reviewed studies published between 2018 (corresponding to BERT’s introduction) and March 2025 that focus on LLM applications for healthcare sentiment analysis with reported performance metrics or qualitative evaluations. Two independent reviewers will screen titles/abstracts and full texts, with disagreements resolved through discussion or third-reviewer consultation. Data extraction will capture study characteristics, research objectives, dataset details, LLM architecture specifications, fine-tuning approaches, performance metrics, and implementation challenges. Quality assessment will employ a modified QUADAS-2 tool and the Cochrane Risk of Bias tool. We will conduct narrative synthesis of the findings, organizing them thematically according to our research questions, with meta-analysis performed if study heterogeneity permits. PROSPERO registration number CRD420251012298 Strengths and limitations of this study This is the first systematic review to comprehensively examine large language models for sentiment analysis specifically within healthcare contexts, addressing a significant gap in the literature. The review’s rigorous methodology follows PRISMA-P guidelines and employs dual independent screening, data extraction, and quality assessment to ensure thoroughness and minimize bias. The inclusion of diverse healthcare text sources (patient feedback, social media, electronic health records) allows for a comprehensive understanding of LLM applications across the healthcare information ecosystem. By focusing on studies published since 2018 (when BERT was introduced), the review captures the most relevant technological developments while excluding outdated approaches. A limitation of this study is the expected heterogeneity across included studies (varying LLM architectures, datasets, metrics, and implementation contexts), which may preclude meaningful meta-analysis and limit definitive conclusions about relative performance, resulting in more descriptive than prescriptive findings." +W7166728658,,10.1192/bjo.2026.11056,2026,Bias Evaluation in Medical Applications of Large Language Models: A Systematic Review,Priya Gupta,5,BJPsych Open,0,"Aims: Large language models (LLMs) are increasingly evaluated and deployed in healthcare, yet unrecognised bias risks reinforcing inequities. This risk is particularly salient in psychiatry, where biased outputs could influence risk formulation, diagnostic reasoning, triage, and therapeutic communication. We examined whether and how bias is evaluated in studies of medical applications of LLMs, and summarised methodologies, reporting practices,and limitations of current bias detection and mitigation approaches to support safe and trustworthy implementation. Methods: We searched databases (EMBASE, MEDLINE, PsycINFO, PubMed, ACL Anthology, ACM Digital Library, arXiv, medRxiv, and bioRxiv; 2017–June 2025), following PRISMA guidance. After de-duplication, >1,800 records were retrieved for title/abstract screening. We assessed 3,664 full-text articles against pre-specified eligibility criteria. Studies were included if they evaluated a medical application of an LLM and reported any explicit bias assessment. Studies that otherwise met eligibility for medical LLM evaluation but did not assess bias were recorded separately to quantify the evidence gap. Screening and data extraction were conducted by multiple reviewers with disagreements resolved by discussion. We extracted specialty (including psychiatry), use case (e.g. decision support, documentation, patient communication), evaluation setting (benchmark/vignette vs clinical data), protected attributes and bias targets assessed, bias methods/metrics, and mitigation strategies proposed or tested. Findings were synthesised narratively with descriptive counts across specialties and use cases. Results: Of 3,664 full texts screened, 278 studies met inclusion by explicitly assessing bias. A further 757 studies evaluated medical applications of LLMs and otherwise met eligibility criteria but did not report any bias assessment, indicating that most medical LLM evaluations omit bias evaluation. Included studies spanned clinical decision support, medical documentation, patient communication, education, and biomedical research across multiple specialties. Bias assessment rates varied by specialty: psychiatry 21/180 (11.7%), radiology 7/107 (6.5%), and oncology 5/51 (9.8%). Bias assessments most commonly examined demographic bias (gender, race/ethnicity, age) and variation in symptom or diseasepresentation. Methodologies included prompt-based probing (e.g. clinical vignettes), counterfactual testing, and red teaming to elicit biased responses; statistical comparisons of performance across groups (including fairness-adjusted performance metrics such as the FAP score); and clinician-led safety evaluation frameworks assessing output suitability. Proposed mitigation strategies included more diverse training data, reinforcement learning with human feedback (RLHF), direct preference optimisation, and ongoing human oversight. Conclusion: Despite rapid growth in medical LLM research, explicit bias assessment remains inconsistent and frequently absent. Standardised, transparent bias evaluation across patient populations, tasks, and clinical contexts is needed to support safe implementation and ensure equitable provision." +W4362522969,,10.4103/singaporemedj.smj-2023-055,2023,The rise of artificial intelligence: addressing the impact of large language models such as ChatGPT on scientific publications,Tiing Leong Ang,4,Singapore Medical Journal,31,"INTRODUCTION Artificial intelligence (AI) permeates our modern society. Chatbots are being used by commercial companies as an interface with their customers, and Internet search engines rely heavily on AI to function. Smart homes, navigation systems and immigration clearance using facial recognition are other examples of the use of AI in our daily lives. In general, AI is an essential element for biotechnology. In biotechnology and research, the applications of AI include drug target identification, image screening, predictive modelling of disease states and outcomes, assessment of functional and structural genomics, metabolomics and proteomics[1] among others. In scientific writing, AI is used in literature search in biomedical databases such as PubMed, and it is present in a rudimentary form in word processing software that researchers use to write scientific manuscripts. AI can be broadly classified into narrow AI, general AI and artificial superintelligence (ASI). Narrow AI is created to solve specific problems. General AI aims to create machines capable of reasoning and thinking like humans. ASI would be capable of outperforming humans. Currently available AI systems are considered narrow AI. LAUNCH OF ChatGPT We entered uncharted territories in scientific writing with the launch of Chat Generative Pre-trained Transformer (ChatGPT; available from: https://chat.openai.com/chat) on 30 November 2022 by OpenAI, an AI research and deployment company. ChatGPT is a large language model (LLM) that has the ability to generate human-like text in a conversational way, including complete paragraphs or even essays, based on specific prompts and context, by mimicking the statistical patterns of language from a huge database of text collated from the Internet. It can potentially write an entire scientific manuscript. Using ChatGPT creates the illusion of conveniently having an expert readily at hand to answer one’s questions at all times. The current version of ChatGPT is a prototype that was trained on dataset that is updated up to 2021. It is not connected to the Internet for real-time updates and does not have access to new knowledge derived after 2021. Therefore, it is to be expected that there will be data gaps in ChatGPT, in terms of new information that becomes available after 2021. In future, the currency of knowledge may be less of an issue if there is continuous input of new data for software training and development, similar to how academics and professionals continuously acquire new knowledge and update themselves. The capabilities of LLMs are limited to the textual training data used, and this can result in problems of reliability and bias. In addition, each LLM has a certain amount of memory, and hence can accept only a finite amount of input to form the context for output generation. There are several issues that need to be properly addressed even at this nascent phase of its development.[2] Concerns have been raised about its role in scientific writing, including whether it can be considered as a co-author, its potential negative impact, such as whether it will result in factual inaccuracies, plagiarism or even outright scientific fraud, and whether it facilitates or blunts the academic development of an individual. Issue of authorship and transparency ChatGPT has been used to draft scientific abstracts and papers, and in the process, it has even been listed as a co-author in academic publications.[3,4,5] The International Committee of Medical Journal Editors (ICMJE) recommends that authorship be based on meeting the following four criteria: (a) substantial contributions to the conception or design of the work, or the acquisition, analysis or interpretation of data for the work; (b) drafting the work or revising it critically for important intellectual content; (c) final approval of the version to be published; and (d) agreement to be accountable for all aspects of the work in ensuring that questions related to the accuracy or integrity of any part of the work are appropriately investigated and resolved.[6] Thus, it is clear that even if ChatGPT or more sophisticated AI programs were used to write the entire manuscript in future, they would still fall short of the eligibility criteria, especially with respect to the issue of personal accountability by authors. AI tools cannot assume such a responsibility. There is consensus by many researchers and journals that AI models cannot be credited as authors,[7,8] and this is the position that the Singapore Medical Journal holds. Transparency and integrity are crucial elements in scientific writing. A submitted academic manuscript must be the original work of the authors, and the wholesale use of ChatGPT without formal acknowledgement is akin to plagiarism. If ChatGPT or any other LLM is to be used as part of the manuscript preparation, there needs to be clarity on how it is being used when the manuscript is written, and whether and how data derived from ChatGPT and other LLMs are further verified. This must be appropriately documented either in the methods or the acknowledgement section of the manuscript. Such transparency is important to prevent erosion of trust when the manuscript is being appraised. It has also been advocated that as an additional safeguard, AI output detectors be incorporated into the editorial process to alert journal editors and reviewers — similar in concept to current approaches of using AI algorithms to screen for plagiarism by checking the similarity index — and to check for bot-like behaviour in the Internet. Tackling the issues of inaccuracies and quality There are concerns about data gaps, factual inaccuracies and bias when ChatGPT is used by authors to draft manuscripts. Although the text written by ChatGPT may appear credible, it has been shown that the generated information can be pure confabulation, with a combination of both facts and fabricated information, or totally fictitious pseudoscientific material. This has been described as artificial hallucination.[9] In future, continuous or regularly scheduled updates of the AI software may help to attenuate this problem of knowledge gaps in the software. Even if there is regular training and updating of the AI software, there may still be a lag phase. Bias in scientific interpretation and writing may also exist, and this could be related to the training of the AI algorithm. It has been suggested that the way the ChatGPT was trained created a left-wing, liberal bias in its responses.[10] The ease of accessing LLMs may result in erosion of skills and expertise needed for high-quality academic publication and promote proliferation of low-quality scientific papers. Therefore, concerns about integrity, accuracy and deterioration of quality with the use of ChatGPT and similar programs in academic writing are justified. However, problems such as academic bias, inaccuracies, ghost authorship, plagiarism and fraud may still arise even in the absence of AI tools like ChatGPT. It is thus crucial that the concept of research integrity be continuously emphasised to all researchers, journals screen and sieve out submissions of dubious quality, and academics and reviewers develop and retain the skillset for critical appraisal, discernment of information and source verification. Clinicians and scientists who read scientific publications must be able to think critically and not accept information at face value. We have already pivoted away from eminence-based medicine to evidence-based medicine. Human verification of information generated by AI remains important. In reality, critical thinking is important not only within academia, but also generally in life, and one needs to be able to separate facts from myths, half-truths, disinformation, pure propaganda and lies. Having insights on how an AI program was developed would allow one to have a better appreciation of its limitation and the potential for bias and errors. A case in point is the knowledge that the current version of ChatGPT has not been trained on new information derived after 2021, and researchers would thus know that ChatGPT cannot be a trusted source for information after 2021. However, a full understanding of the developmental process of the AI model, including the type and extent of information used in its training, and the potential personal bias of people involved in programming may not always be possible, with such systems existing as a black box due to proprietary reasons.[11] Human verification of the accuracy of AI outputs is crucial, and AI should be considered an additional resource to obtain data, with potential for gaps or even inaccuracies, and not a purveyor of absolute truth. CONCEPT OF COLLABORATIVE INTELLIGENCE ChatGPT or other LLMs can facilitate the conduct of research by compiling pre-existing information. For instance, it can be used to draft a study protocol or format the manuscript after completion of the study, which speeds up the writing process. AI can also help improve the language of non-native English speakers and allow translation of an English manuscript to other languages, thus improving access to non-English readers and broadening the reach of journals. However, AI cannot generate new knowledge out of a vacuum when the research has yet to be conducted. Well-trained researchers with deep domain expertise are the ones at the forefront to conduct novel research, generate new findings and achieve scientific breakthroughs. The human interface remains essential for scientific progress. LLMs can be writing assistants and journal assistants but cannot replace human authors or human editors. At the same time, continued training of LLMs will and should occur, so that they can be further improved to help the writing and publishing process. We need to optimise collaborative intelligence, where humans and AI work together, to ensure transparency, integrity and scientific rigour. CONCLUSION AI can be a boon or a bane. While AI is an enabler and a game changer, there exist privacy and ethical concerns, possibility of bias, potential legal liabilities and concerns about validity.[12] In the novel ‘Origin’ written by Dan Brown, a terminally ill Edmond Kirsch instructed Winston, the AI personal assistant he invented, to maximise viewership and hype up an important announcement he was going to make on the Internet. Winston then orchestrated Kirsch’s death during his presentation to make him a martyr, thus boosting online viewership from 3 million to more than 200 million people. AI can facilitate progress, and hence, we should continue to embrace AI and establish collaborative intelligence, especially as it continues to evolve, moving from narrow AI to higher levels of function, such as general AI, or even ASI in future. However, it is important to set up guardrails and a framework to guide its application, so that the potential of AI can be harnessed appropriately." +W4385848697,10.48550/arxiv.2308.06610,10.48550/arxiv.2308.06610,2023,Bio-SIEVE: Exploring Instruction Tuning Large Language Models for Systematic Review Automation,Ambrose Robinson,7,arXiv (Cornell University),9,"Medical systematic reviews can be very costly and resource intensive. We explore how Large Language Models (LLMs) can support and be trained to perform literature screening when provided with a detailed set of selection criteria. Specifically, we instruction tune LLaMA and Guanaco models to perform abstract screening for medical systematic reviews. Our best model, Bio-SIEVE, outperforms both ChatGPT and trained traditional approaches, and generalises better across medical domains. However, there remains the challenge of adapting the model to safety-first scenarios. We also explore the impact of multi-task training with Bio-SIEVE-Multi, including tasks such as PICO extraction and exclusion reasoning, but find that it is unable to match single-task Bio-SIEVE's performance. We see Bio-SIEVE as an important step towards specialising LLMs for the biomedical systematic review process and explore its future developmental opportunities. We release our models, code and a list of DOIs to reconstruct our dataset for reproducibility." +W7127197632,,10.1007/978-3-032-15404-0_5,2026,The Impact of Large Language Models in Healthcare: A Systematic Review of AI Applications,Abhishek Kumar Maurya,3,Lecture notes in networks and systems,0, +W4405750564,,10.1016/j.jval.2024.10.2293,2024,MSR59 Evaluating Systematic Literature Review Screening Performance of Seven Large Language Models in Response to Different Prompting Strategies,Ghayath Janoudi,4,Value in Health,1, +W7104463143,,10.5281/zenodo.17561086,2025,"Large Language Models and ChatGPT in Medical Sciences: Foundations, Capabilities, and Challenges",Amir Tahavvori,27,Zenodo (CERN European Organization for Nuclear Research),0,"Large Language Models (LLMs) such as ChatGPT have transformed how medical knowledge is accessed and used. Built on transformer-based deep learning architectures, these systems are trained on massive datasets containing biomedical and general language sources. This training enables them to interpret clinical text, summarize research findings, and generate coherent, contextually relevant responses. In medicine, LLMs now support a wide range of applications including literature screening, clinical documentation, education, and patient interaction. ChatGPT, in particular, can help synthesize evidence, draft research manuscripts, explain complex medical concepts, and assist in decision support when appropriately guided. These tools have accelerated many aspects of medical research and practice by improving efficiency and accessibility of information. Despite these advances, challenges remain significant. LLMs can produce inaccurate or fabricated information, known as hallucinations, which poses risks in clinical and research settings. Their training data may contain biases that influence outputs, and their decision-making processes are often opaque. Ethical concerns also arise regarding authorship, accountability, and data privacy. To integrate LLMs responsibly, transparency about their use and human oversight are essential. When applied thoughtfully, these tools can enhance scientific discovery, education, and healthcare delivery while maintaining the rigor and ethical standards that underpin medical science." +W7161385519,,10.38124/ijisrt/26apr2488,2026,Autonomous KGs for Large Language Model Agents in Industry: A Systematic Review,Asma Hamad Alharbi,2,International Journal of Innovative Science and Research Technology (IJISRT),0,"The integration of large language models (LLMs) with knowledge graphs (KGs) has emerged as a promising approach to enhance reasoning, explainability, and decision-making in intelligent systems. However, LLMs inherently lack structured knowledge representation, leading to limitations such as hallucinations and reduced reliability, particularly in industrial applications. This study presents a systematic review of autonomous knowledge graphs for LLM-based agents, focusing on their role in enabling knowledge-driven and agentic AI systems in industry. Guided by PRISMA methodology, a comprehensive search across major scientific databases resulted in 33 relevant studies, which were classified into four research dimensions: architectures, KG construction and evolution, KG-based reasoning, and industrial applications. The findings reveal that multi-agent architectures and GraphRAG-based approaches dominate current research, while LLMs enable automated and dynamic knowledge graph lifecycle management. Furthermore, applications in manufacturing, digital twins, and industrial IoT demonstrate significant improvements in efficiency and decision-making. Despite these advancements, challenges related to scalability, computational cost, domain dependency, and lack of real-world validation persist. This review concludes that autonomous knowledge graphs play a critical role in advancing LLM-based agents toward reliable and scalable industrial deployment, while highlighting key research gaps and future directions for developing robust, explainable, and industry-ready AI systems." +W4414078895,,10.1016/j.cont.2025.102246,2025,322 - Role of Large Language Models in Urology: A systematic review and meta-analysis,Mohamed Javid Raja Iyub,11,Continence,1,"Hypothesis / aims of study Large Language Models (LLMs) have demonstrated transformative potential, with promising insights that could potentially enhance clinical practice. Their application in overall healthcare practices is increasingly noted, but their cumulative usage in the field of urology is less explored. Our study aimed to systematically review the current evidence on the usage of LLMs in urology and perform a quantitative analysis of comparable studies. Study design, materials and methods We searched electronic databases namely, Pubmed, Embase, Web of Science, and Scopus to include studies involving urological data or patients in which LLMs were utilized in clinical management or other relevant applications. The studies were grouped based on the application of the LLM that was being studied as: 1) Answering FAQs, 2) Patient Information Materials, 3) Clinical Practice, 4) Medical examination and 5) Other applications. The quality assessment was done utilising the Newcastle-Ottawa Scale. Results A total of 814 articles were found, and after the removal of duplicates and screening of the articles, 39 studies were included in our review. Various applications of LLMs in different domains were listed, and details regarding the training, outcomes, and limitations were studied. ChatGPT was the most commonly studied LLM. The pooled accuracy of the output of various LLMs on various medical examinations was found to be 63.24 (CI 53.69 – 72.72), and a forest plot was constructed. Studies analysing the output of LLMs in answering frequently asked questions reported an accuracy of 67.08% to 100%, with ChatGPT – 4 outperforming ChatGPT – 3.5. Interpretation of results The findings of this systematic review and meta-analysis underscore the growing role of LLMs in urology, highlighting their potential to enhance clinical practice across diverse applications. The pooled accuracy of 63.24% for LLM performance on medical examinations reflects moderate reliability, with variability likely influenced by differences in training data, model architecture, and evaluation methods. Notably, ChatGPT emerged as the most frequently studied LLM, with its latest version (ChatGPT-4) demonstrating superior accuracy (67.08% to 100%) in answering frequently asked questions compared to its predecessor, ChatGPT-3.5. This suggests that advancements in LLM iterations can significantly improve performance, particularly in patient-facing tasks. However, the results also reveal a lack of standardization in evaluation metrics and methodologies across studies, which poses challenges for consistent benchmarking and integration into clinical workflows. Concluding message LLMs' applications are diverse and have augmented urological practice. However, uniform evaluation methods and performance metrics for assessing the output generated for various purposes are needed to further streamline the use and synchronous incorporation of LLM in regular urological practice. Download: Download high-res image (104KB) Download: Download full-size image Figure 1 . Funding None Clinical Trial No Subjects None" +W4415071193,,10.7759/cureus.94300,2025,Performance of ChatGPT and Large Language Models on Medical Licensing Exams Worldwide: A Systematic Review and Network Meta-Analysis With Meta-Regression,Alousious Kasagga,9,Cureus,10,"Large language models (LLMs) are increasingly being tested on national medical licensing examinations, yet existing research is fragmented across models, exam systems, and languages. This study is the first meta-analysis to systematically assess LLM performance across multiple medical licensing exams and languages using pooled estimates, network meta-analysis, and moderator-aware meta-regression. We synthesized accuracy data from 120 evaluations covering 10 exam systems in nine languages, identified through comprehensive searches of PubMed, Web of Science, and Institute of Electrical and Electronics Engineers (IEEE) Xplore, covering the period from 2021 to June 2025. The random-effects meta-analysis showed that 13 of 16 models exceeded the 60% passing threshold, with GPT-o1 leading at 95.4%, followed by DeepSeek-R1 (92.0%) and GPT-4o (89.4%). P-score rankings and network meta-analysis confirmed the superior performance of GPT-o1, while GPT-3.5 and LLaMA-13B consistently underperformed. Meta-regression revealed a significant variation in accuracy by model version, exam system, and language, with lower performance in Chinese and Japanese exams and higher performance in German and Peruvian settings. After adjustment for exam and language, GPT-o1 and DeepSeek-R1 achieved similar accuracy, both significantly higher than GPT-4. Model type, exam system, and language explained most of the between-study heterogeneity (R² =88.99%), and sensitivity analyses supported the robustness of the pooled estimates. Overall, several LLMs now approach or exceed the accuracy required to pass standardized medical licensing exams, supporting their potential role in medical education and decision support." diff --git a/02-literature/search/results/L2-openalex-llm-slr-software-eng.csv b/02-literature/search/results/L2-openalex-llm-slr-software-eng.csv new file mode 100644 index 0000000..358451e --- /dev/null +++ b/02-literature/search/results/L2-openalex-llm-slr-software-eng.csv @@ -0,0 +1,101 @@ +source_id,arxiv,doi,year,title,first_author,n_authors,venue,citations,abstract +W4402665833,,10.1145/3695988,2024,Large Language Models for Software Engineering: A Systematic Literature Review,Xinyi Hou,10,ACM Transactions on Software Engineering and Methodology,1026,"Large Language Models (LLMs) have significantly impacted numerous domains, including Software Engineering (SE). Many recent publications have explored LLMs applied to various SE tasks. Nevertheless, a comprehensive understanding of the application, effects, and possible limitations of LLMs on SE is still in its early stages. To bridge this gap, we conducted a Systematic Literature Review (SLR) on LLM4SE, with a particular focus on understanding how LLMs can be exploited to optimize processes and outcomes. We selected and analyzed 395 research articles from January 2017 to January 2024 to answer four key Research Questions (RQs). In RQ1, we categorize different LLMs that have been employed in SE tasks, characterizing their distinctive features and uses. In RQ2, we analyze the methods used in data collection, pre-processing, and application, highlighting the role of well-curated datasets for successful LLM for SE implementation. RQ3 investigates the strategies employed to optimize and evaluate the performance of LLMs in SE. Finally, RQ4 examines the specific SE tasks where LLMs have shown success to date, illustrating their practical contributions to the field. From the answers to these RQs, we discuss the current state-of-the-art and trends, identifying gaps in existing research, and highlighting promising areas for future study. Our artifacts are publicly available at https://github.com/security-pride/LLM4SE_SLR ." +W4386081573,10.48550/arxiv.2308.10620,10.48550/arxiv.2308.10620,2023,Large Language Models for Software Engineering: A Systematic Literature Review,Xinyi Hou,10,arXiv (Cornell University),114,"Large Language Models (LLMs) have significantly impacted numerous domains, including Software Engineering (SE). Many recent publications have explored LLMs applied to various SE tasks. Nevertheless, a comprehensive understanding of the application, effects, and possible limitations of LLMs on SE is still in its early stages. To bridge this gap, we conducted a systematic literature review (SLR) on LLM4SE, with a particular focus on understanding how LLMs can be exploited to optimize processes and outcomes. We select and analyze 395 research papers from January 2017 to January 2024 to answer four key research questions (RQs). In RQ1, we categorize different LLMs that have been employed in SE tasks, characterizing their distinctive features and uses. In RQ2, we analyze the methods used in data collection, preprocessing, and application, highlighting the role of well-curated datasets for successful LLM for SE implementation. RQ3 investigates the strategies employed to optimize and evaluate the performance of LLMs in SE. Finally, RQ4 examines the specific SE tasks where LLMs have shown success to date, illustrating their practical contributions to the field. From the answers to these RQs, we discuss the current state-of-the-art and trends, identifying gaps in existing research, and flagging promising areas for future study. Our artifacts are publicly available at https://github.com/xinyi-hou/LLM4SE_SLR." +W4414594130,,10.1145/3769676,2025,Large Language Models for Cyber Security: A Systematic Literature Review,Hanxiang Xu,9,ACM Transactions on Software Engineering and Methodology,185,"The rapid advancement of Large Language Models (LLMs) has opened up new opportunities for leveraging artificial intelligence in a variety of application domains, including cybersecurity. As the volume and sophistication of cyber threats continue to grow, there is an increasing need for intelligent systems that can automatically detect vulnerabilities, analyze malware, and respond to attacks. In this survey, we conduct a comprehensive review of the literature on the application of LLMs in cybersecurity (LLM4Security). By comprehensively collecting over 40K relevant papers and systematically analyzing 185 papers from top security and software engineering venues, we aim to provide a holistic view of how LLMs are being used to solve diverse problems across the cybersecurity domain. Through our analysis, we identify several key findings. First, we observe that LLMs are being applied to an expanding range of cybersecurity tasks, including vulnerability detection, malware analysis, and network intrusion detection. Second, we analyze application trends of different LLM architectures (such as encoder-only, encoder-decoder, and decoder-only) across security domains. Third, we identify increasingly sophisticated techniques for adapting LLMs to cybersecurity, such as advanced fine-tuning, prompt engineering, and external augmentation strategies. A significant emerging trend is the use of LLM-based autonomous agents, which represent a paradigm shift from single-task execution to orchestrating complex, multi-step security workflows. Furthermore, we find that the datasets used for training and evaluating LLMs are often limited, highlighting the need for more comprehensive datasets and the use of LLMs for data augmentation. Finally, we discuss the main challenges and opportunities for future research, including the need for more interpretable models, addressing the inherent security risks of LLMs, and their potential for proactive defense. Overall, our survey provides a comprehensive overview of the current state-of-the-art in LLM4Security and identifies several promising directions for future research. We believe that the insights and findings presented in this survey will contribute to the growing body of knowledge on the application of LLMs in cybersecurity and provide valuable guidance for researchers and practitioners working in this field." +W7122824220,,10.5281/zenodo.18215692,2026,Large Language Models for Software Engineering: A Systematic Literature Review,Tahomina Sultana,1,Zenodo (CERN European Organization for Nuclear Research),0, +W4413394768,,10.1109/access.2025.3601206,2025,A Systematic Literature Review of Hallucinations in Large Language Models,Christian Woesle,3,IEEE Access,22,"This review systematically maps research on hallucinations in large language models using a descriptive scheme that links model outputs to four system architectures: unaugmented generation, post-hoc reactive validation, proactive detection-and-mitigation, and fully integrated detection-and-mitigation designs. Our methodology for this systematic review follows the PRISMA guidelines to ensure transparency and reproducibility. We searched IEEE Xplore, ACM Digital Library, and ScienceDirect for studies published between 2015 and January 2025 and extracted 125 peer-reviewed papers across nine application domains. Quantitative analysis shows that question answering and multimodal tasks account for 48% of all papers, whereas software engineering, educational technology, and autonomous systems are underexplored. Although 87.5% of the studies rely on additional reactive or proactive defenses, only 8.8% implement integrated architecture-level safeguards, revealing a critical gap in unified and dynamic architectures. The resulting classification matrix and domain map provide a diagnostic tool for locating blind spots and comparing architectural maturity. Three actionable priorities emerge: develop integrated reasoning-and-verification loops that pre-empt hallucinations; transfer proven causal-intervention and multi-agent validation pipelines to high-stakes, under-represented domains and benchmark them under real conditions; and build modular, cross-domain evaluation frameworks that isolate the contribution of individual mitigation components and support ablation studies. By consolidating fragmented evidence and quantifying architecture-domain imbalances, this review establishes a traceable foundation for engineering reliable, explainable, and domain-adaptable countermeasures to hallucinations in generative language technology." +W4396821716,10.48550/arxiv.2405.04760,10.48550/arxiv.2405.04760,2024,Large Language Models for Cyber Security: A Systematic Literature Review,HanXiang Xu,9,arXiv (Cornell University),27,"The rapid advancement of Large Language Models (LLMs) has opened up new opportunities for leveraging artificial intelligence in a variety of application domains, including cybersecurity. As the volume and sophistication of cyber threats continue to grow, there is an increasing need for intelligent systems that can automatically detect vulnerabilities, analyze malware, and respond to attacks. In this survey, we conduct a comprehensive review of the literature on the application of LLMs in cybersecurity~(LLM4Security). By comprehensively collecting over 40K relevant papers and systematically analyzing 185 papers from top security and software engineering venues, we aim to provide a holistic view of how LLMs are being used to solve diverse problems across the cybersecurity domain. Through our analysis, we identify several key findings. First, we observe that LLMs are being applied to an expanding range of cybersecurity tasks, including vulnerability detection, malware analysis, and network intrusion detection. Second, we analyze application trends of different LLM architectures (such as encoder-only, encoder-decoder, and decoder-only) across security domains. Third, we identify increasingly sophisticated techniques for adapting LLMs to cybersecurity, such as advanced fine-tuning, prompt engineering, and external augmentation strategies. A significant emerging trend is the use of LLM-based autonomous agents, which represent a paradigm shift from single-task execution to orchestrating complex, multi-step security workflows." +W4415088384,10.48550/arxiv.2509.11446,10.48550/arxiv.2509.11446,2025,Large Language Models (LLMs) for Requirements Engineering (RE): A Systematic Literature Review,Mohammad Amin Zadenoori,5,arXiv (Cornell University),9,"Large Language Models (LLMs) are finding applications in numerous domains, and Requirements Engineering (RE) is increasingly benefiting from their capabilities to assist with complex, language-intensive tasks. This paper presents a systematic literature review of 74 primary studies published between 2023 and 2024, examining how LLMs are being applied in RE. The study categorizes the literature according to several dimensions, including publication trends, RE activities, prompting strategies, and evaluation methods. Our findings indicate notable patterns, among which we observe substantial differences compared to previous works leveraging standard Natural Language Processing (NLP) techniques. Most of the studies focus on using LLMs for requirements elicitation and validation, rather than defect detection and classification, which were dominant in the past. Researchers have also broadened their focus and addressed novel tasks, e.g., test generation, exploring the integration of RE with other software engineering (SE) disciplines. Although requirements specifications remain the primary focus, other artifacts are increasingly considered, including issues from issue tracking systems, regulations, and technical manuals. The studies mostly rely on GPT-based models, and often use Zero-shot or Few-shot prompting. They are usually evaluated in controlled environments, with limited use in industry settings and limited integration in complex workflows. Our study outlines important future directions, such as leveraging the potential to expand the influence of RE in SE, exploring less-studied tasks, improving prompting methods, and testing in real-world environments. Our contribution also helps researchers and practitioners use LLMs more effectively in RE, by providing a list of identified tools leveraging LLMs for RE, as well as datasets." +W4408867984,,10.21203/rs.3.rs-5589929/v1,2025,Large Language Model for Requirements Engineering: A Systematic Literature Review,Javed Ali Khan,3,Research Square,14, +W4415333602,10.48550/arxiv.2506.15227,10.48550/arxiv.2506.15227,2025,Large Language Models for Unit Testing: A Systematic Literature Review,Quanjun Zhang,6,arXiv (Cornell University),2,"Unit testing is a fundamental practice in modern software engineering, with the aim of ensuring the correctness, maintainability, and reliability of individual software components. Very recently, with the advances in Large Language Models (LLMs), a rapidly growing body of research has leveraged LLMs to automate various unit testing tasks, demonstrating remarkable performance and significantly reducing manual effort. However, due to ongoing explorations in the LLM-based unit testing field, it is challenging for researchers to understand existing achievements, open challenges, and future opportunities. This paper presents the first systematic literature review on the application of LLMs in unit testing until March 2025. We analyze \numpaper{} relevant papers from the perspectives of both unit testing and LLMs. We first categorize existing unit testing tasks that benefit from LLMs, e.g., test generation and oracle generation. We then discuss several critical aspects of integrating LLMs into unit testing research, including model usage, adaptation strategies, and hybrid approaches. We further summarize key challenges that remain unresolved and outline promising directions to guide future research in this area. Overall, our paper provides a systematic overview of the research landscape to the unit testing community, helping researchers gain a comprehensive understanding of achievements and promote future research. Our artifacts are publicly available at the GitHub repository: https://github.com/iSEngLab/AwesomeLLM4UT." +W7161417267,,10.1145/3815425,2026,A Systematic Literature Review on Detecting Software Vulnerabilities with Large Language Models,Sabrina Kaniewski,5,ACM Transactions on Software Engineering and Methodology,8,"The increasing adoption of Large Language Models (LLMs) in software engineering has sparked interest in their use for software vulnerability detection. However, the rapid development of this field has resulted in a fragmented research landscape, with diverse studies that are difficult to compare due to differences in, e.g., system designs and dataset usage. This fragmentation makes it difficult to obtain a clear overview of the state-of-the-art or compare and categorize studies meaningfully. In this work, we present a comprehensive systematic literature review (SLR) of LLM-based software vulnerability detection. We analyze 263 studies published between January 2020 and November 2025, categorizing them by task formulation, input representation, system architecture, and techniques. Further, we analyze the datasets used, including their characteristics, vulnerability coverage, and diversity. We present a fine-grained taxonomy of vulnerability detection approaches, identify key limitations, and outline actionable future research opportunities. By providing a structured overview of the field, this review improves transparency and serves as a practical guide for researchers and practitioners aiming to conduct more comparable and reproducible research. We publicly release all artifacts and maintain a living repository of LLM-based software vulnerability detection studies at https://github.com/hs-esslingen-it-security/Awesome-LLM4SVD ." +W7167724220,,10.59431/jda.v5i1.810,2026,Large Language Model Integration in Desktop Applications: A Systematic Literature Review,Deyidi Mokoginta,1,Journal Dekstop Application (JDA),0,"The integration of Large Language Models (LLMs) into desktop applications has accelerated sharply over the past several years, yet no prior systematic review has examined this deployment setting as a distinct research domain — separate from web-based or mobile environments, each carrying different architectural constraints, privacy expectations, and user interaction patterns. This study maps, analyzes, and synthesizes the available scientific evidence on LLM architectures adopted in desktop implementations, the technical barriers arising during integration, the measurable effects on User Experience (UX), and the relative effectiveness of competing deployment strategies. A Systematic Literature Review (SLR) was conducted following PRISMA 2020 reporting guidelines, with research questions structured using the PICOC framework. Systematic searches across seven digital databases — IEEE Xplore, ACM Digital Library, Scopus, Web of Science, Google Scholar, arXiv, and Semantic Scholar — covering 2019–2024 yielded 1,200 initial articles, narrowed to 42 final articles through layered selection and quality appraisal using a modified CASP rubric. GPT-4 and the LLaMA family emerged as the dominant architectures, with model selection driven primarily by parameter scale and privacy requirements. Computational constraints, response latency, and data security ranked as the most serious technical barriers, with Retrieval-Augmented Generation (RAG) and model quantization as the most common mitigations. LLM integration was associated with measurable gains in task efficiency and user satisfaction in 73.8% of reviewed studies, though output predictability and user trust remained persistent challenges. A hybrid deployment approach — combining local models with cloud-based API access — emerged as the most common practice among production implementations. Unlike existing reviews addressing LLM capabilities in general software engineering contexts, this study provides the first systematic map of LLM integration specifically within the desktop application domain, offering concrete guidance for developers working at the intersection of artificial intelligence and desktop software engineering." +W7199799894,,,2024,Utilizing Large Language Models to Update Systematic Literature Reviews: A Case Study on Time Pressure and Well-Being in Software Engineering,Jingyi Guo,1,Työväentutkimus Vuosikirja,0,"This thesis examines the application of Large Language Models (LLMs), including ChatGPT-4 and ChatGPT-4.o, to update systematic literature reviews (SLRs) with a focus on time pressure and well-being in software engineering. It aims to address the challenges of traditional SLRs, which often need more timeliness and automaticity in synthesizing the latest research findings. By conducting a case study, this research evaluates the efficiency, comprehensiveness, and accuracy of using LLMs to update SLRs, comparing the strengths and weaknesses of various LLMs in processing and analyzing vast amounts of literature. The study aims to provide insights into the potential of LLMs to revolutionize the methodology of SLRs, particularly in dynamically evolving fields such as software engineering." +W4405640497,,10.1145/3708525,2024,"Efficient and Green Large Language Models for Software Engineering: Literature Review, Vision, and the Road Ahead",Jieke Shi,3,ACM Transactions on Software Engineering and Methodology,43,"Large Language Models (LLMs) have recently shown remarkable capabilities in various software engineering tasks, spurring the rapid growth of the Large Language Models for Software Engineering (LLM4SE) area. However, limited attention has been paid to developing efficient LLM4SE techniques that demand minimal computational cost, time, and memory resources, as well as green LLM4SE solutions that reduce energy consumption, water usage, and carbon emissions. This article aims to redirect the focus of the research community toward the efficiency and greenness of LLM4SE, while also sharing potential research directions to achieve this goal. It commences with a brief overview of the significance of LLM4SE and highlights the need for efficient and green LLM4SE solutions. Subsequently, the article presents a vision for a future where efficient and green LLM4SE revolutionizes the LLM-based software engineering tool landscape, benefiting various stakeholders, including industry, individual practitioners, and society. The article then delineates a roadmap for future research, outlining specific research paths and potential solutions for the research community to pursue. While not intended to be a definitive guide, the article aims to inspire further progress, with the ultimate goal of establishing efficient and green LLM4SE as a central element in the future of software engineering." +W7140887005,,10.1145/3802827,2026,Enhancing Automated Unit Test Generation with Large Language Models: A Systematic Literature Review,Junwei Zhang,5,ACM Transactions on Software Engineering and Methodology,4,"Automated unit test generation is a fundamental yet challenging task in software engineering (SE), playing a critical role in ensuring software correctness, reliability, and maintainability. While traditional approaches such as search-based software testing and symbolic execution have achieved notable success, they often suffer from limited semantic understanding, high configuration costs, and scalability constraints. Recent advances in Large Language Models (LLMs) have fundamentally reshaped the landscape of automated unit testing by enabling models to reason over source code semantics and generate executable, context-aware test cases. Despite the rapid growth of this research area, a comprehensive and task-oriented synthesis of existing work remains lacking. This article presents a systematic literature review of LLM-based unit test generation. This review draws on research from leading SE and AI conferences and journals, including 69 papers published across 25 distinct venues, along with 47 high-quality preprint papers, bringing the total to 116. Our review aims to answer three key research questions: (1) which unit testing tasks have been addressed using LLMs, (2) how LLMs are adapted and integrated into the unit test generation pipeline, and (3) what datasets, benchmarks, and evaluation practices are employed in existing studies. To this end, we organize the literature from a task-centric perspective, covering test generation, test input generation, test oracle generation, and test evolution, and from a methodological perspective, categorizing LLM adaptation strategies into fine-tuning, prompt engineering, and agent-based approaches. Our analysis reveals that current research predominantly focuses on function- and class-level test generation, with comparatively limited attention given to test input generation, oracle construction, and long-term test evolution. Decoder-only LLMs, particularly GPT-family and LLaMA-based models, dominate the field, while encoder-only and encoder–decoder models remain underexplored. We further observe substantial disparities in dataset characteristics, programming language coverage, and evaluation metrics, which hinder fair comparison and reproducibility across studies. Based on empirical evidence extracted from the surveyed literature, we identify key challenges facing LLM-based unit test generation. Building on these findings, we outline several promising research directions, such as dataset optimization, structure-aware context modeling, agent coordination mechanisms, and benchmark enhancement. This review provides a consolidated and evidence-driven foundation for future research, aiming to advance the development of scalable, reliable, and practically applicable LLM-driven unit testing techniques." +W4405543707,,10.1145/3708522,2024,Large Language Model for Vulnerability Detection and Repair: Literature Review and the Road Ahead,Xin Zhou,4,ACM Transactions on Software Engineering and Methodology,137,"The significant advancements in Large Language Models (LLMs) have resulted in their widespread adoption across various tasks within Software Engineering (SE), including vulnerability detection and repair. Numerous studies have investigated the application of LLMs to enhance vulnerability detection and repair tasks. Despite the increasing research interest, there is currently no existing survey that focuses on the utilization of LLMs for vulnerability detection and repair. In this paper, we aim to bridge this gap by offering a systematic literature review of approaches aimed at improving vulnerability detection and repair through the utilization of LLMs. The review encompasses research work from leading SE, AI, and Security conferences and journals, encompassing 43 papers published across 25 distinct venues, along with 15 high-quality preprint papers, bringing the total to 58 papers. By answering three key research questions, we aim to (1) summarize the LLMs employed in the relevant literature, (2) categorize various LLM adaptation techniques in vulnerability detection, and (3) classify various LLM adaptation techniques in vulnerability repair. Based on our findings, we have identified a series of limitations of existing studies. Additionally, we have outlined a roadmap highlighting potential opportunities that we believe are pertinent and crucial for future research endeavors." +W7133294268,10.48550/arxiv.2603.00989,10.48550/arxiv.2603.00989,2026,Energy-Efficient Code Generation Using Large Language Models: A Systematic Literature Review,Sabiya Banu Masthan Ali,5,arXiv (Cornell University),0,"Large Language Models (LLMs) are increasingly used in software engineering for code generation, completion, translation, and repair. While substantial research has examined the energy consumption and environmental impact of LLM training and inference, considerably less attention has been given to the energy efficiency of the code produced by these models. Existing studies are fragmented across different models, programming tasks, benchmarks, hardware platforms, and measurement methods, making it difficult to draw consistent conclusions about the energy efficiency of LLM-generated code. No systematic synthesis currently consolidates these findings or identifies common trends, gaps, and methodological inconsistencies. To address this gap, we conduct a systematic literature review of existing empirical studies on the energy efficiency of LLM-generated code. We examine the extent to which LLMs generate energy-efficient code, the metrics used for evaluation, the benchmarks, datasets, and measurement tools employed, and the influence of prompting and fine-tuning strategies. Our findings show that the current evidence remains limited and highly heterogeneous. Most studies rely on conventional software-efficiency metrics and existing programming or code-generation benchmarks rather than evaluation methods specifically designed for LLM-generated code. The reported energy-efficiency outcomes are strongly dependent on the evaluated model, task, benchmark, and experimental configuration, and no single prompting strategy consistently improves energy efficiency across settings. We further find limited investigation of small language models, energy-aware fine-tuning, diverse programming languages, real-world software domains, and direct hardware-based energy measurement. Overall, the review highlights the lack of standardized evaluation practices and dedicated energy-efficiency benchmarks." +W7133570722,,,2026,Energy-Efficient Code Generation Using Large Language Models: A Systematic Literature Review,Sabiya Banu Masthan Ali,5,arXiv (Cornell University),0,"Large Language Models (LLMs) are increasingly used in software engineering for code generation, completion, translation, and repair. While substantial research has examined the energy consumption and environmental impact of LLM training and inference, considerably less attention has been given to the energy efficiency of the code produced by these models. Existing studies are fragmented across different models, programming tasks, benchmarks, hardware platforms, and measurement methods, making it difficult to draw consistent conclusions about the energy efficiency of LLM-generated code. No systematic synthesis currently consolidates these findings or identifies common trends, gaps, and methodological inconsistencies. To address this gap, we conduct a systematic literature review of existing empirical studies on the energy efficiency of LLM-generated code. We examine the extent to which LLMs generate energy-efficient code, the metrics used for evaluation, the benchmarks, datasets, and measurement tools employed, and the influence of prompting and fine-tuning strategies. Our findings show that the current evidence remains limited and highly heterogeneous. Most studies rely on conventional software-efficiency metrics and existing programming or code-generation benchmarks rather than evaluation methods specifically designed for LLM-generated code. The reported energy-efficiency outcomes are strongly dependent on the evaluated model, task, benchmark, and experimental configuration, and no single prompting strategy consistently improves energy efficiency across settings. We further find limited investigation of small language models, energy-aware fine-tuning, diverse programming languages, real-world software domains, and direct hardware-based energy measurement. Overall, the review highlights the lack of standardized evaluation practices and dedicated energy-efficiency benchmarks." +W7214983381,,10.1016/j.jss.2026.113129,2026,Large language models in software project management: A two-phase systematic literature review,Muhammad Saad Amin,4,Journal of Systems and Software,0,"Background Large Language Models (LLMs) are increasingly integrated into Software Engineering (SE) processes and practices. However, their application within Software Project Management (SPM) remains fragmented and insufficiently synthesized. This absence of systematic evidence prevents both researchers and practitioners from understanding where LLMs can meaningfully contribute to project management processes in relation to established standards such as the Project Management Body of Knowledge (PMBOK). Objectives This study aims to systematically synthesize existing evidence on the application and integration of LLMs in SPM by means of a two-phase method where Phase I systematically searches for existing secondary studies in the topic of interest, and Phase II (whenever applicable) carries out an SLR in the topic of interest. Method We conducted a two-phase method: Phase I is a tertiary study that used 14 research questions to assess the existence, coverage, and quality of prior secondary studies in LLMs applied to SPM. As no suitable SLR was identified from Phase I, we also conducted Phase II, which employed the SEGRESS guidelines to analyze primary studies that empirically evaluated the use of LLMs in SPM via five research questions. Phase II also assessed coverage across PMBOK processes and synthesized reported mechanisms, outcomes, research designs and evaluation approaches. Results The SLR (Phase II) shows that studies employ encoder-only and decoder-only LLM architectures in equal proportion (17 studies each). Their applications focus on a narrow set of SPM tasks, including effort estimation, project planning, and requirements-related activities. Coverage across PMBOK processes is uneven, with limited attention to areas such as procurement, stakeholder, quality, and cost management. Most studies rely on proof-of-concept evaluations, with scarce industrial validation and heterogeneous evaluation metrics. Conclusion The results indicate that current research on LLMs in SPM provides limited guidance for systematic adoption. One of the main drawbacks for such lack of guidance is the weak empirical validation in many of the included primary studies, limited PMBOK coverage, and lack of context-aware integration of LLMs in SPM. 1" +W7166589016,,10.63317/5dnbkz4d44kz,2026,Automatic Generation of Discharge Summaries Using Large Language Models: A Systematic Literature Review,Lucas Molino-Piñar,3,Language Resources and Evaluation Conference (LREC),0,"Discharge summaries are critical documents for continuity of care, yet their manual creation imposes significant burdens on clinical staff.This systematic literature review examines current approaches to automatic generation of discharge summaries using Natural Language Processing (NLP) and Large Language Models (LLMs).Following the Kitchenham guidelines for systematic reviews in software engineering, we searched Scopus and PubMed databases for studies published between 2023 and 2026, identifying 9 primary studies from an initial pool of 102 papers.Our analysis reveals that GPT-4 and its variants dominate current research (appearing in 6 of 9 studies), while open-source alternatives like LLaMA show promise for privacy-preserving deployments.Evaluation primarily relies on automatic metrics (ROUGE, BLEU) combined with human expert assessment.Key challenges include hallucination rates ranging from 33% to 64%, information omission, integration with Electronic Health Record (EHR) systems, and context window limitations.Studies addressing factuality employ human-in-the-loop validation, prompt engineering techniques, and knowledge graph-based correction mechanisms.Despite these challenges, recent implementations demonstrate clinical feasibility, with one study achieving a 94.35% System Usability Score.This review provides a comprehensive synthesis of the state-of-the-art and identifies opportunities for future research in this rapidly evolving field." +W7125135101,,10.1145/3747588,2025,A Survey on Large Language Models for Code Generation,Juyong Jiang,5,ACM Transactions on Software Engineering and Methodology,248,"Large Language Models (LLMs) have garnered remarkable advancements across diverse code-related tasks, known as Code LLMs, particularly in code generation that generates source code with LLM from natural language descriptions. This burgeoning field has captured significant interest from both academic researchers and industry professionals due to its practical significance in software development, e.g., GitHub Copilot . Despite the active exploration of LLMs for a variety of code tasks, either from the perspective of Natural Language Processing (NLP) or Software Engineering (SE) or both, there is a noticeable absence of a comprehensive and up-to-date literature review dedicated to LLM for code generation. In this survey, we aim to bridge this gap by providing a systematic literature review that serves as a valuable reference for researchers investigating the cutting-edge progress in LLMs for code generation. We introduce a taxonomy to categorize and discuss the recent developments in LLMs for code generation, covering aspects such as data curation, latest advances, performance evaluation, ethical implications, environmental impact, and real-world applications. In addition, we present a historical overview of the evolution of LLMs for code generation and provide a quantitative and qualitative comparative analysis of experimental results of code LLMs, sourced from their original papers to ensure a fair comparison on the HumanEval, MBPP, and BigCodeBench benchmarks, across various levels of difficulty and types of programming tasks, to highlight the progressive enhancements in LLM capabilities for code generation. We identify critical challenges and promising opportunities regarding the gap between academia and practical development. Furthermore, we have established a dedicated resource GitHub page ( https://github.com/juyongjiang/CodeLLMSurvey ) to continuously document and disseminate the most recent advances in the field." +W7165734347,,10.21203/rs.3.rs-10126259/v1,2026,Are Large Language Models Ready for Quantum Software Engineering? A Multivocal Literature Review,Antonio Della Porta,4,Research Square,0, +W7130568179,,10.1109/fllm67465.2025.11390990,2025,Towards the Integration of Large Language Models into the Software Development Life Cycle: A Systematic Literature Review,Mohammadamin Madani,6,International Conference on Foundation and Large Language Models (FLLM),1,"Models (LLMs) are reshaping the Software Development Life Cycle (SDLC) by introducing new levels of automation and intelligent assistance across core phases, including requirements engineering, design, testing, and maintenance. Previous works have examined the adoption of LLM in these areas, but have overlooked its influence on traditional practices in SDLC. Therefore, this paper builds upon this foundation and investigates, from a phase-level perspective, how LLMs interact with the SDLC as a whole and whether their presence necessitates rethinking of conventional methodologies. To achieve this objective, this article reviews the state of the field using a systematic literature review method. Peer-reviewed studies published between 2019 and 2025, examining the integration, benefits, and limitations of LLM within SDLC are analyzed. The findings indicate that LLMs are most commonly applied in implementation and testing, where they improve productivity through code generation, test automation, and fault localisation. However, their adoption also introduces challenges related to model hallucination, contextual misalignment, computational overhead, and ethical risks such as bias and opacity. Therefore, we synthesise several LLM integration patterns and stress various risk mitigation strategies that can be adopted to enhance the integration of LLMs into software engineering practices. The study concludes by outlining practical implications for software teams and proposing future research directions to improve LLM’s integration with agile, DevOps and CI/CD workflows." +W7132830204,,10.6084/m9.figshare.30042556,2026,Are Large Language Models Ready for Quantum Software Engineering? A Multivocal Literature Review,Antonio Della Porta,1,Figshare,0,"This repository contains all the articles screened and included in our multivocal literature review on the use of Large Language Models in Quantum Software Engineering. The data stored in the repository was collected through queries conducted on ACM Digital Library, IEEE Xplore, Scopus, arXiv, and Google Search. The repository also reports the inclusion, exclusion, and quality-assessment criteria applied throughout the study selection process." +W4394653832,10.48550/arxiv.2404.04566,10.48550/arxiv.2404.04566,2024,"Efficient and Green Large Language Models for Software Engineering: Literature Review, Vision, and the Road Ahead",Jieke Shi,3,arXiv (Cornell University),4,"Large Language Models (LLMs) have recently shown remarkable capabilities in various software engineering tasks, spurring the rapid growth of the Large Language Models for Software Engineering (LLM4SE) area. However, limited attention has been paid to developing efficient LLM4SE techniques that demand minimal computational cost, time, and memory resources, as well as green LLM4SE solutions that reduce energy consumption, water usage, and carbon emissions. This paper aims to redirect the focus of the research community towards the efficiency and greenness of LLM4SE, while also sharing potential research directions to achieve this goal. It commences with a brief overview of the significance of LLM4SE and highlights the need for efficient and green LLM4SE solutions. Subsequently, the paper presents a vision for a future where efficient and green LLM4SE revolutionizes the LLM-based software engineering tool landscape, benefiting various stakeholders, including industry, individual practitioners, and society. The paper then delineates a roadmap for future research, outlining specific research paths and potential solutions for the research community to pursue. While not intended to be a definitive guide, the paper aims to inspire further progress, with the ultimate goal of establishing efficient and green LLM4SE as a central element in the future of software engineering." +W4401879229,,10.1109/compsac61105.2024.00096,2024,Systematic Literature Review of Prompt Engineering Patterns in Software Engineering,Yuya Sasaki,6,IEEE Annual International Computer Software and Applications Conference (COMPSAC),23,"Advancements in large language models (LLMs) are transforming software engineering through innovative prompt engineering strategies. By analyzing prompt-driven enhancements across key software engineering tasks, we present a sys-tematic literature review and a pioneering taxonomy elucidating the practical applications of prompt engineering in software engineering. Our taxonomy offers a foundational framework that clarifies the roles of prompt engineering and measures its impact, thereby guiding evolving AI -driven software engineering research and practices." +W7159734530,,,2026,LLM-Assisted Empirical Software Engineering: Systematic Literature Review and Research Agenda,Victória Gomes,5,arXiv (Cornell University),0,"Context: Empirical Software Engineering (ESE) faces increasing challenges due to data scale, methodological complexity, and reproducibility concerns. Large Language Models (LLMs) have emerged as promising tools to support empirical workflows, yet their use remains fragmented, with no comprehensive synthesis to guide responsible adoption. Aims: This study analyzes how LLMs are used in ESE, examining supported tasks, phases of the empirical lifecycle, integration into workflows, reported benefits and limitations, and the extent of reproducibility-related reporting. It also identifies gaps and future research directions. Method: We conducted a systematic literature review of peer-reviewed papers (2020-2025) across 12 leading software engineering venues, resulting in 50 primary studies analyzed through qualitative and quantitative synthesis. Results: We identified 69 LLM-assisted tasks, mainly in mining software repositories and controlled experiments, focusing on classification, filtering, and evaluation. LLMs are used across multiple phases but are concentrated in data processing and analysis. Their integration is largely automation-oriented, with limited decision-support use. Benefits emphasize efficiency and scalability, while limitations include hallucinations, inconsistency, prompt sensitivity, and reproducibility issues. Reporting practices are often incomplete. Conclusion: LLM use in ESE is growing but remains automation-driven, with gaps in human-centered integration and transparency. We outline implications and research agenda for responsible use." +W7131521417,,,2025,"Efficient and green large language models for software engineering: Literature review, vision, and the road ahead",Jieke Shi,3,Singapore Management University Institutional Knowledge (InK) (Singapore Management University),0,Singapore National Research Foundation (NRF) +W7158894149,10.48550/arxiv.2604.26192,10.48550/arxiv.2604.26192,2026,LLM-Assisted Empirical Software Engineering: Systematic Literature Review and Research Agenda,Victória Gomes,5,arXiv (Cornell University),0,"Context: Empirical Software Engineering (ESE) faces increasing challenges due to data scale, methodological complexity, and reproducibility concerns. Large Language Models (LLMs) have emerged as promising tools to support empirical workflows, yet their use remains fragmented, with no comprehensive synthesis to guide responsible adoption. Aims: This study analyzes how LLMs are used in ESE, examining supported tasks, phases of the empirical lifecycle, integration into workflows, reported benefits and limitations, and the extent of reproducibility-related reporting. It also identifies gaps and future research directions. Method: We conducted a systematic literature review of peer-reviewed papers (2020-2025) across 12 leading software engineering venues, resulting in 50 primary studies analyzed through qualitative and quantitative synthesis. Results: We identified 69 LLM-assisted tasks, mainly in mining software repositories and controlled experiments, focusing on classification, filtering, and evaluation. LLMs are used across multiple phases but are concentrated in data processing and analysis. Their integration is largely automation-oriented, with limited decision-support use. Benefits emphasize efficiency and scalability, while limitations include hallucinations, inconsistency, prompt sensitivity, and reproducibility issues. Reporting practices are often incomplete. Conclusion: LLM use in ESE is growing but remains automation-driven, with gaps in human-centered integration and transparency. We outline implications and research agenda for responsible use." +W4412164231,,10.1109/wsese66602.2025.00010,2025,On the Difficulties of Conducting and Replicating Systematic Literature Reviews Studies Using LLMs in Software Engineering,Katia Romero Felizardo,7,IEEE/ACM International Workshop on Methodological Issues with Empirical Studies in Software Engineering (WSESE),6,"The Software Engineering (SE) community has adopted Systematic Literature Reviews (SLRs) to summarize the state-of-the-art in specific research topics. SLRs offer benefits such as synthesizing evidence from diverse studies to generate auditable results following a reproducible approach, and identifying research gaps for future exploration. However, the process is effort-intensive, prone to errors, and lays various challenges during their conduction. To overcome some of these issues, there is a growing belief that Large Language Models (LLMs) can support systematic literature reviews. While the literature has shown promising results in social sciences, more evidence of its accuracy is needed in technical fields like SE. In this context, studies and replications are essential in verifying the benefits and drawbacks of applying LLMs in systematic literature reviews. This paper discusses the difficulties in conducting and replicating studies that adopt LLMs to support systematic literature in SE. As an implication, we identified the challenges of adopting LLM in SLRs and offered a list of open issues for future research." +W4403413467,,10.1145/3674805.3686666,2024,ChatGPT application in Systematic Literature Reviews in Software Engineering: an evaluation of its accuracy to support the selection activity,Katia Romero Felizardo,5,Proceedings of the ... ACM-IEEE International Symposium on Empirical Software Engineering and Measurement/International Symposium on Empirical Software Engineering and Measurement,28,"Context: The Systematic Literature Review (SLR) process involves searching, selecting, and synthesizing relevant literature on a specific research topic for evidence-based decision-making in Software Engineering (SE). Due to the time-consuming of the SLR process, tool support is essential. Gap: ChatGPT is a significant advancement in Natural Language Processing (NLP), and it can potentially accelerate time-consuming and propone-error activities, such as the selection activity of the SLR process. Therefore, having a tool to assist in the selection process appears beneficial, and we argue that ChatGPT can facilitate the analysis of extensive studies, saving time and effort. Objective: We aim to evaluate the accuracy (i.e., studies correctly classified) of using ChatGPT–4.0 in SLR in SE, particularly to support the first stage, based on the title, abstract, and keywords. Method: We assessed the accuracy of utilizing ChatGPT for selecting studies, the first stage, to be included in two SLRs (SLR1 and SLR2), in contrast to the conventional method of reading the title and abstract. Results: The accuracy of ChatGPT supporting the initial selection activity was 75.3% (SLR1 – 101 correct selections: 48 inclusions and 53 exclusions; 33 incorrect selections: 17 inclusions and 16 exclusions) and 86.1% (SLR2 – 386 correct selections: 113 inclusions and 273 exclusions; 62 incorrect selections: 27 inclusions and 35 exclusions). Conclusions: Our accuracy results indicate that it is not advisable to completely outsource the selection process to ChatGPT. However, it could be valuable as a support tool, aiding novice researchers or even experienced ones when they are in doubt." +W7164174508,,10.5281/zenodo.20481599,2026,Dataset for Requirements Analysis in Software Engineering Using Large Language Models,Daniel Mongeote-Tlachy,3,Zenodo (CERN European Organization for Nuclear Research),0,"This dataset supports the transparency and reproducibility of the systematic literature review and procedures reported in the article ""Requirements Analysis in Software Engineering Using Large Language Models.""The folder ""01-Need for a SLR search results"" documents the initial automated search process conducted in the Scopus and Web of Science databases. It includes the retrieved records based on the filters applied during the preliminary review of related literature.The folder ""02-Automatic search results"" contains the subsequent search stages carried out across selected digital library databases (IEEE Xplore, ACM Digital Library, ScienceDirect, and Scopus). It includes exported bibliographic records organized by search strings, as well as the inclusion and exclusion steps applied.The folder ""03-Atlas.TI"" contains all the data used and generated by the cualitatite data manager software.The file ""Dataset.xlsx"" serves as the main dataset. It contains fully curated data from the entire review process, including metadata for the selected studies: titles, venues, publication year, AI techniques, LLMs used, prompting strategies, evaluation methods, reported benefits, reported limitations and research gaps identified. It also includes raw and normalized quality assessment scores, aligned with the Kitchenham's methodology for software engineering research.Together, these materials enable full reproducibility of the review and provide a structured foundation for future research in AI-assisted web accessibility." +W7123360572,,10.1109/access.2026.3652325,2026,Methods and Techniques of Agentic Software Engineering: A Systematic Literature Review,Nesreen A. Otoum,2,IEEE Access,6,"The development of agentic software engineering (ASE) is shifting the software development process, specifically by adopting autonomous agents into the fundamental software development process, where autonomous agents using large language models (LLMs) and artificial intelligence (AI) make decisions, write code, and collaborate across the entire software development lifecycle. The systematic review is a summary of literature on the agentic software engineering methodologies, frameworks, and techniques published in 2022-2025. We identify and compare methods based on autonomous coding, multi-agent systems, iterative refinement, and human-agent collaboration, drawing on 61 high-quality studies. We find that LLM-based agents show substantial potential in automated software engineering and that recent work is exploring new trends in agent cooperation, reasoning systems, and tool integration. We recognize such key challenges as the coordination procedures, reliability, explainability, and the ethical issues. This review characterizes the emerging role of agentic software engineering in improving the productivity of developers, the quality of their code, and the automation of their processes, and it offers a structured roadmap for future research. With the rapid evolution of the field, agentic approaches will tend to transform the conception, creation, and maintenance of software and provide more autonomous, adaptive, and intelligent systems." +W7111070211,,10.5753/jserd.2025.5097,2025,Natural Language Processing in Software Engineering: A Systematic Literature Review,Gabriel Nogueira Pacheco,5,Journal of Software Engineering Research and Development,0,"Context: Software engineering (SE) artifacts and documents, such as requirements specifications, user stories, test cases, and concepts of operations (ConOps), are typically written in natural language, making their manipulation challenging. Natural Language Processing (NLP) is a viable solution for managing these tasks. Objective: To conduct a systematic literature review to explore the current use of NLP in SE artifacts and tasks, supplemented by a tertiary study focusing on the emerging role of Large Language Models (LLMs) in software engineering research. Method: We searched digital libraries for relevant papers and applied inclusion and exclusion criteria to filter the primary studies. We then analyzed NLP techniques applied to SE documents and examined their usage in this context. Our research methodology followed Kitchenham and Charters' guidelines. Additionally, we conducted a tertiary study to synthesize findings from existing systematic literature reviews and surveys specifically addressing LLMs in software engineering. Results: We selected 60 primary studies to identify the most common methods for NLP pipelines, feature extraction, language models, and machine learning algorithms used in SE. We also assessed the purposes of these methods, their benefits for SE, their difficulty, and their contribution to SE advancement. The tertiary study revealed a rapid proliferation of LLM-focused research, with comprehensive reviews documenting exponential growth in publications and widespread adoption across diverse SE tasks. Conclusion: Requirements are the most frequently addressed artifacts using NLP techniques, with preprocessing and part-of-speech (POS) tagging being widely used. There is a notable increase in the use of large language models for various SE tasks, such as requirements elicitation, source code generation, bug fixing, and software testing. The tertiary study confirms that LLMs represent a pivotal shift in the research landscape, warranting dedicated investigation to understand their transformative impact on NLP applications in software engineering." +W4415064809,,10.1016/j.jss.2025.112758,2025,A systematic literature review of software engineering research on Jupyter notebook,Md. Saeed Siddik,3,Journal of Systems and Software,3,"• This research provides the first comprehensive systematic literature review on software engineering research specifically targeting Jupyter notebooks, identifying 199 primary studies published up to September 2025 and categorizing them into 11 core software engineering topics. • This research reveals that a large portion of the studies have been published outside traditional software engineering venues, with Human-Computer Interaction conferences like ACM Conference on Human Factors in Computing Systems (CHI) being the top publishing venues, highlighting the interdisciplinary nature of Jupyter Notebook research. • This research identifies a reusability gap in existing research, showing that only 82 out of 199 studies offer usable replication packages, and most are hosted on GitHub instead of permanent repositories, which violates open science best practices. • This research identifies that notebook-specific solutions for software engineering issues such as testing, refactoring, and documentation are relatively underexplored. Future directions include resolving duplicated execution numbers, refactoring inter-notebook clones, and generating grouped documentation for coherent-code cells are future directions derived from our study. • This research proposes the integration of modern AI-based solutions into Jupyter notebooks to support various software engineering topics, including code search and code generation. Additionally, future research should leverage advanced AI techniques (e.g., large language models), to improve conversational AI-powered assistants for automated code generation by multi-step workflow automation in data science notebooks. • Although the paper exceeds the recommended length due to the inclusion of detailed tables, figures, and categorized analyses (covering 11 topics and 21 subtopics), we believe that this extended content is essential for clearly and completely reporting our findings. As the first systematic literature review in this domain, we have carefully structured the paper to ensure readability. We believe the length is justified by the value and breadth of this paper’s contributions. Context : Jupyter Notebook has emerged as a versatile tool that transforms how researchers, developers, and data scientists conduct and communicate their work. As the adoption of Jupyter notebooks continues to rise, so does the interest from the software engineering research community in improving the software engineering practices for Jupyter notebooks. Objective : The purpose of this study is to analyze trends, gaps, and methodologies used in software engineering research on Jupyter notebooks. Method : We selected 199 relevant publications up to September 2025, following established systematic literature review guidelines. We explored publication trends, categorized them based on software engineering topics, and reported findings based on those topics. Results : The most popular venues for publishing software engineering research on Jupyter notebooks are related to human-computer interaction instead of traditional software engineering venues. Researchers have addressed a wide range of software engineering topics on notebooks, such as code reuse, readability, and execution environment. Although reusability is one of the research topics for Jupyter notebooks, only 82 of the 199 studies can be reused based on their provided URLs. Additionally, most replication packages are not hosted on permanent repositories for long-term availability and adherence to open science principles. Conclusion : Solutions specific to notebooks for software engineering issues, including testing, refactoring, and documentation, are underexplored. Future research opportunities exist in automatic testing frameworks, refactoring clones between notebooks, and generating group documentation for coherent code cells." +W7134020591,,,2026,Large Language Models In High-Level Software Testing : A Literature Review,Toni Grönberg,1,Tampere University Institutional Repository (Tampere University),0,"Large language models (LLMs) have rapidly become influential tools in software engineering, raising new possibilities for supporting software testing across multiple levels of abstraction. Early applications of LLMs have focused strongly on low-level, code-centric tasks such as unit test generation, code repair, and test oracle inference. In contrast, high-level testing activities—including system, integration, acceptance, security, and GUI testing—have received considerably less attention despite their central role in validating the behaviour, reliability, and quality of complex software systems. This thesis investigates how LLMs are currently utilised in high-level software testing and examines the extent to which emerging research addresses the challenges inherent to these tasks. The thesis begins by introducing the foundations of software testing, defining test levels relevant to high-level dynamic testing, and outlining the essential properties of modern LLMs that enable their use in testing workflows. It then examines the traditional use of LLMs in testing, highlighting the limitations of early prompt-based approaches that relied on narrow context windows, limited autonomy, and weak integration with execution environments. Building on this, the thesis explores the rise of LLM-based testing agents capable of iterative reasoning, tool use, and adaptive test generation within feedback-driven loops. These agentic approaches represent a major shift from static prompt usage toward more autonomous testing systems that can interact with complex software artefacts. The analysis of high-level application areas demonstrates that LLM-supported methods are emerging in security testing, system testing, GUI exploration, acceptance testing, requirements analysis, and migration testing. Across these domains, LLMs contribute by interpreting naturallanguage specifications, generating structured scenarios, analysing outputs, and navigating interfaces in ways that traditional scripted automation struggles to achieve. At the same time, recurring limitations—such as hallucination, reproducibility issues, integration challenges, and inconsistent evaluation practices—highlight that LLM-based testing remains an evolving area with significant room for maturation. The thesis concludes that while high-level software testing has only recently begun to benefit from LLM-supported techniques, the field is progressing rapidly through the development of agent-based systems and hybrid architectures. LLMs are not yet reliable enough to replace human expertise in high-level testing, but they increasingly serve as capable collaborators that can accelerate test design, extend automation into previously inaccessible tasks, and support exploratory analysis. These findings suggest that continued research, better evaluation standards, and domain-informed model grounding are essential for advancing the practical adoption of LLMs in high-level software testing." +W7135074624,,10.1109/sami68106.2026.11420586,2026,Large Language Models in Software Documentation and Modeling: A Literature Review and Findings,Lukáš Radoský,2,IEEE World Symposium on Applied Machine Intelligence and Informatics (SAMI),1,"Generative artificial intelligence attracts significant attention, especially with the introduction of large language models. Its capabilities are being exploited to solve various software engineering tasks. Thanks to their ability to understand natural language and generate natural language responses, large language models are great for processing various software documentation artifacts. At the same time, large language models excel at understanding structured languages, having the potential for working with software programs and models. We conduct a literature review on the usage of large language models for software engineering tasks related to documentation and modeling. We analyze articles from four major venues in the area, organize them per tasks they solve, and provide an overview of used prompt techniques, metrics, approaches to human-based evaluation, and major datasets." +W7171820009,10.48550/arxiv.2607.26100,10.48550/arxiv.2607.26100,2026,Large Language Models for Software Engineering Diagrams: A Systematic Review of UML and ER modelling,Mojdeh Rahmanian,3,arXiv (Cornell University),0,"Large language models (LLMs) are increasingly applied to diagram-based software and data modelling. Among various modelling notations, UML and entity-relationship (ER) diagrams are the most widely adopted for software modelling and data modelling, respectively. Recent literature has investigated various applications of LLMs in diagram modelling; however, their effectiveness and limitations have not been extensively discussed. This systematic literature review analyses 64 studies published between 2023 and 2025, examining diagram coverage, modelling tasks, technical approaches, evaluation practices, and limitations. Our findings reveal significant concentration patterns and gaps. UML-based software modelling strongly dominates, with class diagrams receiving the most attention whilst behavioural diagrams and data modelling remain underrepresented. Diagram construction from natural language is the primary focus, with limited work on transformation, quality assurance, and consistency checking. GPT-based models are heavily prevalent, raising concerns about reproducibility and vendor dependence. Evaluation practices are heterogeneous, employing diverse metrics and custom datasets with limited benchmark reuse and inconsistent reporting of robustness and statistical significance. Common limitations include semantic inaccuracies, hallucinated diagram elements, sensitivity to prompt formulation, and reproducibility constraints. This survey provides the first systematic synthesis of LLM-based diagram modelling research, highlighting needs for standardised benchmarks, stronger evaluation protocols, broader diagram coverage, and techniques for improving semantic reliability and multi-view consistency." +W7172068650,,,2026,Large Language Models for Software Engineering Diagrams: A Systematic Review of UML and ER modelling,Mojdeh Rahmanian,3,arXiv (Cornell University),0,"Large language models (LLMs) are increasingly applied to diagram-based software and data modelling. Among various modelling notations, UML and entity-relationship (ER) diagrams are the most widely adopted for software modelling and data modelling, respectively. Recent literature has investigated various applications of LLMs in diagram modelling; however, their effectiveness and limitations have not been extensively discussed. This systematic literature review analyses 64 studies published between 2023 and 2025, examining diagram coverage, modelling tasks, technical approaches, evaluation practices, and limitations. Our findings reveal significant concentration patterns and gaps. UML-based software modelling strongly dominates, with class diagrams receiving the most attention whilst behavioural diagrams and data modelling remain underrepresented. Diagram construction from natural language is the primary focus, with limited work on transformation, quality assurance, and consistency checking. GPT-based models are heavily prevalent, raising concerns about reproducibility and vendor dependence. Evaluation practices are heterogeneous, employing diverse metrics and custom datasets with limited benchmark reuse and inconsistent reporting of robustness and statistical significance. Common limitations include semantic inaccuracies, hallucinated diagram elements, sensitivity to prompt formulation, and reproducibility constraints. This survey provides the first systematic synthesis of LLM-based diagram modelling research, highlighting needs for standardised benchmarks, stronger evaluation protocols, broader diagram coverage, and techniques for improving semantic reliability and multi-view consistency." +W7128096177,,,2026,Large Language Models in Software Documentation and Modeling: A Literature Review and Findings,Lukáš Radoský,2,arXiv (Cornell University),0,"Generative artificial intelligence attracts significant attention, especially with the introduction of large language models. Its capabilities are being exploited to solve various software engineering tasks. Thanks to their ability to understand natural language and generate natural language responses, large language models are great for processing various software documentation artifacts. At the same time, large language models excel at understanding structured languages, having the potential for working with software programs and models. We conduct a literature review on the usage of large language models for software engineering tasks related to documentation and modeling. We analyze articles from four major venues in the area, organize them per tasks they solve, and provide an overview of used prompt techniques, metrics, approaches to human-based evaluation, and major datasets." +W7128002728,10.48550/arxiv.2602.04938,10.48550/arxiv.2602.04938,2026,Large Language Models in Software Documentation and Modeling: A Literature Review and Findings,Lukáš Radoský,2,arXiv (Cornell University),0,"Generative artificial intelligence attracts significant attention, especially with the introduction of large language models. Its capabilities are being exploited to solve various software engineering tasks. Thanks to their ability to understand natural language and generate natural language responses, large language models are great for processing various software documentation artifacts. At the same time, large language models excel at understanding structured languages, having the potential for working with software programs and models. We conduct a literature review on the usage of large language models for software engineering tasks related to documentation and modeling. We analyze articles from four major venues in the area, organize them per tasks they solve, and provide an overview of used prompt techniques, metrics, approaches to human-based evaluation, and major datasets." +W4416231638,,10.1145/3841636,2026,Large Language Model for Verilog Code Generation: Literature Review and the Road Ahead,Guang Yang,17,ACM Computing Surveys,2,"Code generation represents a critical intersection of Software Engineering (SE) and Artificial Intelligence (AI). Within this broader landscape, Verilog, as a representative hardware description language (HDL), is fundamental to Electronic Design Automation (EDA), recent research has increasingly focused on leveraging Large Language Models (LLMs) to automate Verilog code generation, particularly at the Register Transfer Level (RTL) design. Despite growing interest, a comprehensive survey of this domain remains absent. This review addresses this gap by providing a systematic literature review of LLM-based Verilog code generation, analyzing 102 papers (70 published and 32 high-quality preprints) from SE, AI, and EDA venues. We structure our analysis around four key research questions: (1) identifying the LLMs utilized, (2) examining evaluation datasets and metrics, (3) categorizing generation techniques, and (4) analyzing alignment approaches. Furthermore, we synthesize findings to identify critical limitations in current studies regarding effectiveness and integration. Finally, we outline a roadmap highlighting potential opportunities for future research in LLM-assisted hardware design." +W4416452806,10.48550/arxiv.2505.16697,10.48550/arxiv.2505.16697,2025,Software Architecture Meets LLMs: A Systematic Literature Review,Larissa Schmid,8,arXiv (Cornell University),4,"Large Language Models (LLMs) are used for many different software engineering tasks. In software architecture, they have been applied to tasks such as classification of design decisions, detection of design patterns, and generation of software architecture design from requirements. However, there is little overview on how well they work, what challenges exist, and what open problems remain. In this paper, we present a systematic literature review on the use of LLMs in software architecture. We analyze 18 research articles to answer five research questions, such as which software architecture tasks LLMs are used for, how much automation they provide, which models and techniques are used, and how these approaches are evaluated. Our findings show that while LLMs are increasingly applied to a variety of software architecture tasks and often outperform baselines, some areas, such as generating source code from architectural design, cloud-native computing and architecture, and checking conformance remain underexplored. Although current approaches mostly use simple prompting techniques, we identify a growing research interest in refining LLM-based approaches by integrating advanced techniques." +W4393217463,,10.1007/978-981-97-1711-8_12,2024,A Pilot Study on AI-Assisted Code Generation with Large Language Models for Software Engineering,Hsiao-Chuan Liu,3,Communications in computer and information science,4, +W4413788802,,10.3390/ai6090206,2025,Unlocking the Potential of the Prompt Engineering Paradigm in Software Engineering: A Systematic Literature Review,Irdina Wanda Syahputri,3,AI,8,"Prompt engineering (PE) has emerged as a transformative paradigm in software engineering (SE), leveraging large language models (LLMs) to support a wide range of SE tasks, including code generation, bug detection, and software traceability. This study conducts a systematic literature review (SLR) combined with a co-citation network analysis of 42 peer-reviewed journal articles to map key research themes, commonly applied PE methods, and evaluation metrics in the SE domain. The results reveal four prominent research clusters: manual prompt crafting, retrieval-augmented generation, chain-of-thought prompting, and automated prompt tuning. These approaches demonstrate notable progress, often matching or surpassing traditional fine-tuning methods in terms of adaptability and computational efficiency. Interdisciplinary collaboration among experts in AI, machine learning, and software engineering is identified as a key driver of innovation. However, several research gaps remain, including the absence of standardized evaluation protocols, sensitivity to prompt brittleness, and challenges in scalability across diverse SE applications. To address these issues, a modular prompt engineering framework is proposed, integrating human-in-the-loop design, automated prompt optimization, and version control mechanisms. Additionally, a conceptual pipeline is introduced to support domain adaptation and cross-domain generalization. Finally, a strategic research roadmap is presented, emphasizing future work on interpretability, fairness, and collaborative development platforms. This study offers a comprehensive foundation and practical insights to advance prompt engineering research tailored to the complex and evolving needs of software engineering." +W4406325768,,10.1145/3712003,2025,"LLM-Based Multi-Agent Systems for Software Engineering: Literature Review, Vision, and the Road Ahead",Junda He,3,ACM Transactions on Software Engineering and Methodology,246,"Integrating Large Language Models (LLMs) into autonomous agents marks a significant shift in the research landscape by offering cognitive abilities that are competitive with human planning and reasoning. This article explores the transformative potential of integrating Large Language Models into Multi-Agent (LMA) systems for addressing complex challenges in software engineering (SE). By leveraging the collaborative and specialized abilities of multiple agents, LMA systems enable autonomous problem-solving, improve robustness, and provide scalable solutions for managing the complexity of real-world software projects. In this article, we conduct a systematic review of recent primary studies to map the current landscape of LMA applications across various stages of the software development lifecycle (SDLC). To illustrate current capabilities and limitations, we perform two case studies to demonstrate the effectiveness of state-of-the-art LMA frameworks. Additionally, we identify critical research gaps and propose a comprehensive research agenda focused on enhancing individual agent capabilities and optimizing agent synergy. Our work outlines a forward-looking vision for developing fully autonomous, scalable, and trustworthy LMA systems, laying the foundation for the evolution of Software Engineering 2.0." +W7203792166,,10.1038/s41598-026-65738-4,2026,A maturity model for integrating large language models to address key challenges in software engineering education,Abdulrahman Alzahrani,2,Scientific Reports,0,"The integration of Large Language Models (LLM) into Software Engineering Education (SEE) holds significant potential to address key challenges such as diversity of students, rapid technological changes and resource constraints. This paper provides an extensive multivocal maturity model for the use of LLM in SEE, developed through multivocal literature review (MLR), an empirical survey and an interpretive structural modeling (ISM). The suggested model outlines and assesses the critical process areas (PAs) essential for incorporating LLM into SE curricula and provides a systematic way to improve educational practices. The study finds five maturity levels in SEE issues, such as Initial, Managed, Defined, Quantitatively Managed and Optimizing, related to student engagement, technological development, skills based on practical experiences and ethical issues. A mixed method approach (theoretical and practical) will be employed in exploring the potential of the LLMs for personalised learning, real-time feedback and interactive pedagogies. Teachers and institutions to determine their readiness level to implement LLM in SEE and continuously improve it for further innovation use the outcomes as a reference. The research is a contribution to the expanding research on how to leverage AI in the education and offers a novel framework to pursue the use of LLMs in the educational context of software engineering." +W7116968648,,10.5335/rbca.v17i3.16310,2025,A Systematic Review about Large Language Models (LLMs) applied to Code Generation,Fabio Alecsandro Bacin,4,Revista Brasileira de Computação Aplicada,1,"Large Language Models (LLMs) for code generation represent a significant advancement in software development by boosting productivity, simplifying repetitive tasks, enabling automated testing, and promoting best practices. This paper presents a systematic literature review (SLR) of studies focused on LLMs applied to code generation. The review enhances understanding of the capabilities and limitations of these models, outlining both their benefits and challenges. The review protocol included a search on the Google Scholar database using relevant keywords related to LLMs and code generation. A total of 112 articles were initially retrieved, from which 15 were selected based on relevance and quality criteria. Of these, 8 were analyzed in depth to evaluate various approaches and outcomes, while the remaining 7 provided the theoretical foundation for the study. This work contributes to the growing body of knowledge in the field and supports future research and applications of LLMs in software engineering." +W4414276043,,10.5753/ise.2025.14892,2025,Investigating the use of Large Language Models in software security requirements: Results of a literature review,Amália Melo,3,,1,"This study investigates the application of Large Language Models (LLMs) in the context of security requirements engineering through the conduction of a rapid literature review. The review enabled the characterization of current research in this domain with respect to: (i) the purposes for which LLMs are employed in security requirements activities; (ii) the families of LLMs explored (e.g., GPT, BERT, LLaMA), their capabilities (e.g., classification, generation), and underlying architectures (e.g., encoder, decoder, encoder-decoder); (iii) the techniques adopted for conditioning or guiding LLM behavior; (iv) the datasets used to train, fine-tune, or validate these models; and (v) the evaluation metrics applied to assess the performance of LLMs in supporting security requirements tasks. The findings contribute to a structured understanding of the current state of research and highlight key trends, gaps, and opportunities for advancing the use of LLMs in secure software engineering." +W4405522167,,10.1109/conisoft63288.2024.00050,2024,State of the Art of the Security of Code Generated by LLMs: A Systematic Literature Review,Leonardo Criollo Ramírez,4,International Conference in Software Engineering Research and Innovation (CONISOFT),9,"AI -assisted programming has experienced a surge in popularity over the past few years, largely thanks to advancements in Large Language Model technologies. This has led to the emergence of tools like ChatGPT and GitHub Copilot. However, the use of AI models for code generation comes with a downside: the resulting code is susceptible to vulnerabilities, thus posing new challenges in the field of secure software development. In this study, we analyze the current state of research regarding the security of LLM generated code from the Software Engineering perspective. We conducted a Systematic Literature Review following the guidelines from Kitchenham et al. The search process included five sources: IEEE Xplore, ACM, Science Direct, Springer Link and Wiley Online Library. We also included an iteration of backward and forward snowballing. We obtained 3104 peer-reviewed studies though Quasi-Gold aided automated search and selected the most relevant ones through 5 stages. The final selection includes 15 primary studies from which we extracted and synthesized data. We identified seven different kinds of security vulnerability present in LLM generated code, six different mitigation strategies and practices, and four tools recommended by authors to use in conjunction with LLM code generation. Security related issues within LLM generated code have only just begun to be explored, initial research has already emphasized the significance of considering the inclusion of AI-powered code generation in software projects, as it carries the risk of introducing vulnerabilities at a higher rate than human-generated code. The vulnerabilities, practices and tools identified in this study, can potentially help developers to use LLM programming assistants more responsibly, making informed decisions when leveraging LLM technology." +W4416418710,,10.1016/j.infsof.2025.107960,2025,Using LLMs to enhance code quality: A systematic literature review,Nawaf Alomari,4,Information and Software Technology,4, +W7162402178,,10.20944/preprints202605.1638.v1,2026,A Systematic Literature Review on Generative AI in Software Engineering: Code Generation and Refactoring,Rafshan Bin Razzak,1,Preprints.org,0,"Generative AI has emerged as a transformative force in software engineering, particularly in code generation and refactoring, yet a comprehensive synthesis of its applications, challenges, and future directions remains lacking. This systematic literature review aims to consolidate existing research on generative AI in software engineering, focusing on its role in automating code generation, optimizing refactoring processes, and addressing broader implications such as ethics, security, and human-AI collaboration. We systematically analyze peer-reviewed studies to identify key trends, methodologies, and gaps in the literature, then categorize findings into dimensions including software development processes, testing, education, and interdisciplinary challenges. The review reveals that generative AI significantly accelerates code production and improves refactoring efficiency, but its adoption is constrained by issues such as code quality assurance, ethical concerns, and the need for effective human oversight. While advancements in large language models have expanded capabilities, their integration into real-world software engineering workflows requires further empirical validation. The study also highlights the growing importance of AI-assisted education and collaborative tools, suggesting that future research should prioritize transparency, robustness, and domain-specific adaptations. By synthesizing diverse perspectives, this review provides a foundational framework for researchers and practitioners to navigate the evolving landscape of generative AI in software engineering, while identifying critical areas for innovation and responsible deployment." +W4406890462,,10.1109/access.2025.3535782,2025,Time-Series Large Language Models: A Systematic Review of State-of-the-Art,Shamsu Abdullahi,6,IEEE Access,41,"Large Language Models (LLMs) have transformed Natural Language Processing (NLP) and Software Engineering by fostering innovation, streamlining processes, and enabling data-driven decision-making. Recently, the adoption of LLMs in time-series analysis has catalyzed the emergence of time-series LLMs, a rapidly evolving research area. Existing reviews provide foundational insights into time-series LLMs but lack a comprehensive examination of recent advancements and do not adequately address critical challenges in this domain. This Systematic Literature Review (SLR) bridges these gaps by analysing state-of-the-art contributions in time-series LLMs, focusing on architectural innovations, tokenisation strategies, tasks, datasets, evaluation metrics, and unresolved challenges. Using a rigorous methodology based on PRISMA guidelines, over 700 studies from 2020 to 2024 were reviewed, with 59 relevant studies selected from journals, conferences, and workshops. Key findings reveal advancements in architectures and novel tokenization strategies tailored for temporal data. Forecasting dominates the identified tasks with 79.66% of the selected studies, while classification and anomaly detection remain underexplored. Furthermore, the analysis reveals a strong reliance on datasets from the energy and transportation domains, highlighting the need for more diverse datasets. Despite these advancements, significant challenges persist, including tokenization inefficiencies, prediction hallucinations, and difficulties in modelling long-term dependencies. These issues hinder the robustness, scalability, and adaptability of time-series LLMs across diverse applications. To address these challenges, this SLR outlines a research roadmap emphasizing the improvement of tokenization methods, the development of mechanisms for capturing long-term dependencies, the mitigation of hallucination effects, and the design of scalable, interpretable models for diverse time-series tasks." +W7163994597,,10.46799/ajesh.v5i6.779,2026,Large Language Models for Intelligent Code Generation in Software Engineering: A Systematic Review and Future Research Directions,Ahmad Wali Noori,2,Asian Journal of Engineering Social and Health,0,"The proliferation of Large Language Models has catalyzed a paradigm shift in software engineering automation, yet existing literature reviews predominantly evaluate functional correctness while systematically neglecting security posture, maintainability, and multi-language deployment contexts. This study addresses the critical research gap regarding the absence of unified, multi-dimensional assessment protocols for LLM-generated code in production environments. Through a PRISMA-guided systematic review of 87 primary studies published between 2020 and 2025, this research examined architectural evolution, evaluation methodologies, and deployment barriers using a dual-phase qualitative and quantitative synthesis supported by a custom quality assessment framework. A five-criterion evaluation instrument was applied to ensure methodological rigor, with strong inter-rater reliability (Cohen’s kappa = 0.84). The findings reveal that decoder-only transformer architectures have achieved dominant performance on generative benchmarks, with Claude-3 Opus attaining 74.9% Pass@1 on HumanEval. However, only 12% of evaluated studies incorporated security metrics, fewer than 8% assessed maintainability, and benchmark contamination threatens the validity of reported generalization. The novelty of this work lies in proposing a unified evaluation framework that integrates functional correctness, Common Weakness Enumeration vulnerability scanning, and maintainability metrics, alongside a standardized multi-language protocol. The implications suggest that enterprise DevSecOps pipelines must embed static security analysis and cross-language quality gates before production integration. Future research should prioritize runtime verification, longitudinal productivity studies, and contamination-free benchmarks to ensure trustworthy deployment of generative AI in mission-critical software ecosystems." +W4399431381,,10.1145/3643915.3644088,2024,Exploring the Potential of Large Language Models in Self-adaptive Systems,Jialong Li,6,International Symposium on Software Engineering for Adaptive and Self-Managing Systems (SEAMS),17,"Large Language Models (LLMs), with their abilities in knowledge acquisition and reasoning, can potentially enhance the various aspects of Self-adaptive Systems (SAS). Yet, the potential of LLMs in SAS remains largely unexplored and ambiguous, due to the lack of literature from flagship conferences or journals in the field, such as SEAMS and TAAS. The interdisciplinary nature of SAS suggests that drawing and integrating ideas from related fields, such as software engineering and autonomous agents, could unveil innovative research directions for LLMs within SAS. To this end, this paper reports the results of a literature review of studies in relevant fields, summarizes and classifies the studies relevant to SAS, and outlines their potential to specific aspects of SAS. Literature classification: www.github.com/545659928/LLM4SAS" +W7214166792,,10.5281/zenodo.22802731,2026,Supplementary Material: Impostor Phenomenon and Generative AI in Software Engineering -- A Systematic Literature Review,Jonathan Felipe Hurtado Diaz,3,Zenodo (CERN European Organization for Nuclear Research),0,"FILE DESCRIPTIONS 00_protocol_slr.pdf Full SLR protocol (version 2.0, 2026-05-13) in Spanish. Includes research questions, PICOC+F framework, inclusion/exclusion criteria (CI1-CI6, CE1-CE7), screening procedure, MMAT quality assessment plan, data extraction strategy (28 fields), and synthesis approach. Based on Kitchenham & Charters (2007) and PRISMA 2020 (Page et al., 2021). Note: CI2 was expanded on 2026-06-06 (director approval) to include adjacent constructs (self-efficacy, cognitive load, psychological ownership, intrinsic motivation, stress). 00_protocol_slr_en.pdf English translation of the full SLR protocol (version 2.0). 01_screening_decisions_F2.xlsx Phase 2 (full-text) screening decisions for all 57 studies evaluated at full text. Includes inclusion/exclusion verdict with textual evidence cited from each paper (section, page). Dual reviewer (R1 + R2) with arbitration by thesis director (R3) for disagreements. 27 excluded with documented reasons; 9 not retrieved with access attempts documented. 02_data_extraction_matrix.xlsx Data extraction matrix (v2) with 28 fields across 8 categories for all 24 included publications (23 studies). Categories: metadata, study design, population/demographics, IP measurement, GenAI interaction, productivity/team impact, interventions, limitations. E0216 and E0235 are extracted separately (different constructs reported) but counted as one study. 03_mmat_quality_scores.xlsx Mixed Methods Appraisal Tool (MMAT 2018) quality profiles for all 23 included studies. Each study scored on 2 screening items + 5 design-specific items as Yes/No/NPD (no global score, per MMAT 2018 guidelines). Dual evaluation by R1 and R2; inter-rater agreement: kappa = 0.49 (moderate, N=145 items). E0224 excluded post-extraction (MMAT annulled). 04_snowballing_tracking.xlsx Forward and backward snowballing tracking (Wohlin, 2014) applied to 14 seed studies across 3 tiers. Two directed expansion passes via Semantic Scholar (May-July 2026). 36 unique candidates evaluated; 4 incorporated (E0401-E0404). Second pass yielded one additional study (E0404). The traversal was not exhaustive: only two of 14 planned seeds and two secondary seeds were traversed. The 25 exclusions made solely by R0 were validated by R1 (25/25 confirmed, 0 discrepancies). The notas_r1 column contains the justification for each validation. Additional sheet ""Validacion R0"" with summary by exclusion criterion. 05_screening_decisions_F1.xlsx Phase 1 (title + abstract) consolidated screening decisions for all 269 unique records after deduplication. Includes AI agent (R0) advisory classification and independent human reviewer decisions (R1, R2). R1 and R2 independently evaluated 61 records; 54 agreements (88.5%); binary kappa (Include+Dudoso vs Exclude) = 0.80. R0 coincided with the final human classification in 109/262 records (41.6%; kappa = 0.114); R0 classified 234/269 as ""Dudoso"" (87.0%). 06_screening_agent.py Python script for the LLM-based screening agent (Llama 3.3 70B via Groq API). Advisory role only (R0); human reviewers made all final inclusion/exclusion decisions independently. Classifies records against CI/CE criteria and assigns relevance scores. The agent classified 87% of records as ""Dudoso"" (uncertain); its recall for inclusion was 100% (zero false negatives). 07_search_strings_final.txt Complete search strings (v4.2) for all 6 databases (Scopus, Web of Science, IEEE Xplore, EBSCO/PsycINFO, SpringerLink, ScienceDirect). Two complementary sub-chains: Foundation-A (IP x SE population) and GenAI-B (IP + adjacent constructs x SE x GenAI). Includes database-specific syntax adaptations. 08_kappa_calculation.py Python script for computing Cohen's kappa inter-rater agreement between human reviewers (R1 and R2). Supports both F1 screening and MMAT quality assessment agreement calculations. 09_exclusions_extraction_phase.md Decisions made after F2 screening during full-text extraction: (1) E0224 excluded for not measuring the impostor phenomenon; (2) E0239 included as adjacent construct (relative self-efficacy); (3) E0216/E0235 treated as one study with two publications. Documents the rationale and PRISMA implications for each decision. 10_screening_agent_procedure.md Detailed procedure for the AI-assisted screening (section 4.3): how each reviewer used the screening agent outputs, independence protocol, and the separation between R0 (advisory) and R1/R2 (decision) roles. 11_exclusiones_snowballing.md Resolution of the 6 snowballing candidates that remained as ""Dudoso"" after R0 screening. All 6 were reviewed by R1 on 2026-09-15 and excluded: 4 failed CI2 (construct not in the taxative list) and 2 failed CE4 (non-empirical/WIP). Includes rationale per case and final snowballing counts. 12_veredictos_consenso_mmat.pdf MMAT 2018 consensus verdicts document. Contains the agreed-upon quality profiles after three rounds of disagreement resolution between R1 and R2, verifying against original publications." +W7133610361,,10.1145/3800580,2026,Novice Developers’ Perspectives on Adopting LLMs for Software Development: A Systematic Literature Review,Samuel Ferino,4,ACM Transactions on Software Engineering and Methodology,5,"Following the rise of large language models (LLMs), many studies have emerged in recent years focusing on exploring the adoption of LLM-based tools for software development by novice developers: computer science/software engineering students and early-career industry developers with two years or less of professional experience. These studies have sought to understand the perspectives of novice developers on using these tools, a critical aspect of the successful adoption of LLMs in software engineering. To systematically collect and summarise these studies, we conducted a systematic literature review (SLR) following the guidelines by Kitchenham et al. on 80 primary studies published between April 2022 and June 2025 to answer four research questions (RQs). In answering RQ1, we categorised the study motivations and methodological approaches. In RQ2, we identified the software development tasks for which novice developers use LLMs. In RQ3, we categorised the advantages, challenges, and recommendations discussed in the studies. Finally, we discuss the study limitations and future research needs suggested in the primary studies in answering RQ4. Throughout the paper, we also indicate directions for future work and implications for software engineering researchers, educators, and developers. Our research artifacts are publicly available at https://github.com/Samuellucas97/SupplementaryInfoPackage-SLR ." +W4416545253,10.48550/arxiv.2503.07556,10.48550/arxiv.2503.07556,2025,Novice Developers' Perspectives on Adopting LLMs for Software Development: A Systematic Literature Review,Samuel Ferino,4,arXiv (Cornell University),1,"Following the rise of large language models (LLMs), many studies have emerged in recent years focusing on exploring the adoption of LLM-based tools for software development by novice developers: computer science/software engineering students and early-career industry developers with two years or less of professional experience. These studies have sought to understand the perspectives of novice developers on using these tools, a critical aspect of the successful adoption of LLMs in software engineering. To systematically collect and summarise these studies, we conducted a systematic literature review (SLR) following the guidelines by Kitchenham et al. on 80 primary studies published between April 2022 and June 2025 to answer four research questions (RQs). In answering RQ1, we categorised the study motivations and methodological approaches. In RQ2, we identified the software development tasks for which novice developers use LLMs. In RQ3, we categorised the advantages, challenges, and recommendations discussed in the studies. Finally, we discuss the study limitations and future research needs suggested in the primary studies in answering RQ4. Throughout the paper, we also indicate directions for future work and implications for software engineering researchers, educators, and developers. Our research artifacts are publicly available at https://github.com/Samuellucas97/SupplementaryInfoPackage-SLR." +W7156306625,,10.22214/ijraset.2026.80859,2026,Analyzing Human-AI Collaborative Experience in Modern Software Development Using Large Language Models (Literature Review),Aman Singh,1,International Journal for Research in Applied Science and Engineering Technology,0,"The rise of Large Language Models which are often referred to as LLMs has brought about a significant change in how software development is conducted. This change marks the start of a collaborative relationship between humans and artificial intelligence. LLMs are not just automation tools anymore; they are increasingly seen as intelligent partners that help developers during various stages of the software development process. This literature review seeks to analyze critically the nature of collaboration between humans and AI in modern software engineering, focusing specifically on how LLMs impact developer experience, productivity, and decision-making. This paper draws on a wide array of academic studies, technical reports, and industry practices to look into the various roles that LLMs play in tasks like code generation, debugging, documentation, testing, and knowledge retrieval. It examines how developers engage with these systems in real-world situations, often participating in iterative and conversational workflows that mix human creativity with machine assistance. The review points out that LLMs not only speed up development timelines but also change cognitive workflows by lessening repetitive tasks and allowing developers to concentrate on more complex problem-solving and design issues. However, the use of LLMs in software development comes with its own set of challenges. This paper evaluates critically the concerns regarding the reliability and accuracy of outputs generated by these models, the danger of becoming too reliant on AI systems, and the possible decline of basic programming skills. It also discusses wider issues such as ethical concerns, data privacy, biases in the outputs from models, and the lack of transparency in decisions made by AI." +W7147690639,,10.5281/zenodo.19368366,2026,LLMs on code smell detection and refactoring: a systematic literature review [Replication Package],Mateus Dias,2,Zenodo (CERN European Organization for Nuclear Research),0,Replication Package of research submitted to Journal of Software Engineering Research and Development +W7130607038,,10.1145/3797276,2026,"LLM-as-a-Judge for Software Engineering: Literature Review, Vision, and the Road Ahead",Junda He,8,ACM Transactions on Software Engineering and Methodology,12,"The rapid integration of Large Language Models (LLMs) into software engineering (SE) has revolutionized tasks from code generation to program repair, producing a massive volume of software artifacts. This surge in automated creation has exposed a critical bottleneck: the lack of scalable and reliable methods to evaluate the quality of these outputs. Human evaluation, while effective, is very costly and time-consuming. Traditional automated metrics like BLEU rely on high-quality references and struggle to capture nuanced aspects of software quality, such as readability and usefulness. In response, the LLM-as-a-Judge paradigm, which employs LLMs for automated evaluation, has emerged. This approach leverages the advanced reasoning and coding capabilities of LLMs themselves to perform automated evaluations, offering a compelling path toward achieving both the nuance of human insight and the scalability of automated systems. Nevertheless, LLM-as-a-Judge research in the SE community is still in its early stages, with many breakthroughs needed. This forward-looking SE 2030 paper aims to steer the research community toward advancing LLM-as-a-Judge for evaluating LLM-generated software artifacts, while also sharing potential research paths to achieve this goal. We provide a literature review of existing SE studies on LLM-as-a-Judge and envision these frameworks as reliable, robust, and scalable human surrogates capable of evaluating software artifacts with consistent, multi-faceted assessments by 2030 and beyond. To validate this vision, we analyze the limitations of current studies, identify key research gaps, and outline a detailed roadmap to guide future developments of LLM-as-a-Judge in software engineering. While not intended to be a definitive guide, our work aims to foster further research and adoption of LLM-as-a-Judge frameworks within the SE community, ultimately improving the effectiveness and scalability of software artifact evaluation methods." +W7162322444,,10.5281/zenodo.20387265,2026,(Artifact) Ai LitReview - Web-Based Tool for LLM-Assisted Systematic Literature Reviews,Anderson Deizepe,1,Zenodo (CERN European Organization for Nuclear Research),0,"AI LitReview is a collaborative web-based platform designed to support researchers throughout the entire Systematic Literature Review (SLR) process, from protocol definition to evidence synthesis. The tool integrates Large Language Models (LLMs) via the OpenRouter API (with access to over 365 models) and locally via Ollama, enabling AI-assisted study selection, structured data extraction, and RAG-based report generation. Key features include: configurable prompt templates with dynamic variables, multi-model screening comparison, full-text PDF screening, per-study cost and token tracking, human-in-the-loop decision-making, multi-user collaboration, Sankey-based Study Flow Diagram, and XLS export at each phase. This record contains the source code, demo video, and supplementary materials for the paper ""AI LitReview: A Web-Based Tool for AI-Assisted Systematic Literature Reviews"", submitted to the 40th Brazilian Symposium on Software Engineering (SBES 2026). The tool is publicly accessible at https://ailitreview.dzp.net.br and released under the MIT license." +W4392781018,10.48550/arxiv.2403.07506,10.48550/arxiv.2403.07506,2024,"Robustness, Security, Privacy, Explainability, Efficiency, and Usability of Large Language Models for Code",Zhou Yang,5,arXiv (Cornell University),16,"Large language models for code (LLM4Code), which demonstrate strong performance (e.g., high accuracy) in processing source code, have significantly transformed software engineering. Many studies separately investigate the non-functional properties of LM4Code, but there is no systematic review of how these properties are evaluated and enhanced. This paper fills this gap by thoroughly examining 146 relevant studies, thereby presenting the first systematic literature review to identify seven important properties beyond accuracy, including robustness, security, privacy, explainability, efficiency, and usability. We discuss the current state-of-the-art methods and trends, identify gaps in existing research, and present promising directions for future study." +W7125834859,,10.1109/tse.2026.3657900,2026,Agentic Recommender Systems: A Systematic Literature Review,Ivens Da Silva Portugal,3,IEEE Transactions on Software Engineering,2,"Recommender systems (RSs) are software systems that use machine learning techniques to suggest items, such as movies, products or routes to users based on input provided over time. These systems have been widely used by companies, such as Amazon, Google, and Netflix, who rely heavily on recommendations as a core element of their interaction with users. In the case of software engineering, RSs are used to recommend, for example, software tasks, developers, APIs, libraries, and bug fixes. More recently, advancements in natural language processing and large language models (LLMs) have led to the creation of LLM agents capable of generating plans, interacting with users and the environment, receiving feedback, and using tools. These developments have resulted in the use of LLM agents in RSs, giving rise to multi-agent RSs referred to as agentic recommender systems (ARSs). However, agentic RSs are not yet well characterized in terms of the basic elements of system modeling and design. This systematic literature review characterizes agentic recommender systems with respect to agents, roles, relationships, prompts, integration, use cases, strategies and evaluation methods. An analysis of published studies yields insights, such as (i) the identification of 13 distinct types of agents, (ii) the popularity of GPT models, (iii) a framework for ARSs, (iv) a generalized prompt, (v) the use of persona, cues, output format, and zero-shot prompting, (vi) the frequent use of Amazon, Yelp, and MovieLens datasets, (vii) the use of nDCG@k and Recall@k as common evaluation metrics, and (viii) the identification of many research gaps. This systematic literature review aims to support the continued research and development of agentic recommender systems." +W7223032718,,10.5281/zenodo.23498699,2026,"Ambiguity in LLM-Based UML Generation: A Systematic Literature Review of Errors, Mitigation, and Validation",Driss Rami,5,Zenodo (CERN European Organization for Nuclear Research),0,"# Protocol and Replication Package for: Ambiguity in LLM-Based UML Generation: A Systematic Literature Review of Errors, Mitigation, and Validation This deposit contains the review protocol, the search strategies, the screening logs, the extracted dataset, the quality assessment scores, and the analysis scripts supporting the systematic literature review (SLR) reported in the manuscript: > Rami, D., El Bahri, A., Zougari, O., Kharmoum, N., & Ziti, S. *Ambiguity in LLM-Based UML Generation: A Systematic Literature Review of Errors, Mitigation, and Validation.* (2026). Manuscript under review. ## Review scope The review synthesizes 62 primary studies published between January 2023 and December 2025 on the use of large language models (LLMs) for transforming natural-language software requirements into Unified Modeling Language (UML) models. It addresses ten primary research questions (RQ1–RQ10) and four secondary review questions (SRQ1–SRQ4), covering: - requirement types transformed into UML models;- UML diagram types targeted;- LLM families, versions, and adaptation strategies;- validation mechanisms (syntactic, structural, semantic, traceability);- datasets and benchmarks;- evaluation metrics;- error and hallucination patterns;- reproducibility artifacts;- research gaps and a research agenda. ## Methodology The review follows the PRISMA 2020 statement (Page et al., 2021) and the Kitchenham & Charters (2007) guidelines for software engineering systematic reviews. Primary studies were retrieved from Scopus, Web of Science, IEEE Xplore, and the ACM Digital Library, using the database-specific queries documented in `search_strings/`. The full review protocol is provided in `protocol.pdf`. ## Contents of this deposit | File / folder | Description ||---|---|| `protocol.pdf` | Full review protocol: research questions, eligibility criteria (IC1–IC6, EC1–EC6), search strategy, screening procedure, data extraction form, quality assessment checklist (QA1–QA10), synthesis plan. || `search_strings/` | One text file per database (`scopus.txt`, `wos.txt`, `ieee.txt`, `acm_abstract.txt`, `acm_title.txt`) with the exact query, execution date, filters, and number of records retrieved. || `screening_logs/` | Title/abstract and full-text screening decisions with study identifiers and exclusion reasons (EC1–EC6). || `extraction_dataset.xlsx` | The 62 included primary studies coded across all fields of the extraction form (identification, requirement type, UML diagram type, LLM family/version, adaptation strategy, validation layer, metrics, error categories, reproducibility artifacts). || `quality_assessment.xlsx` | QA1–QA10 scores per included study, with total score and quality band (High / Moderate / Low). || `scripts/` | Python scripts used for deduplication (DOI, normalized title, fuzzy Jaccard matching ≥ 0.90), automatic keyword pre-screening, descriptive statistics, and figure generation. || `README.md` | This file. | ## Software environment - Python 3.11- pandas, matplotlib, rapidfuzz, openpyxl- Deduplication threshold: Jaccard similarity ≥ 0.90 on normalized titles- The scripts are provided as-is and were run on the dataset included in this deposit. ## How to cite If you use the protocol or the extraction dataset, please cite both the manuscript and this deposit: > Rami, D., El Bahri, A., Zougari, O., Kharmoum, N., & Ziti, S. (2026). *Protocol and replication package for: Ambiguity in LLM-Based UML Generation: A Systematic Literature Review*. Zenodo. https://doi.org/10.5281/zenodo.XXXXXXX ## License The protocol, dataset, and scripts are released under the Creative Commons Attribution 4.0 International license (CC BY 4.0). You are free to share and adapt the material for any purpose, provided that appropriate credit is given. ## Contact Driss Rami — drami.cpr@gmail.com — Intelligent Processing and Security of Systems (IPSS) Laboratory, Faculty of Sciences, Mohammed V University, Rabat, Morocco." +W7114889968,,10.3390/bdcc9120320,2025,"A Systematic Literature Review of Retrieval-Augmented Generation: Techniques, Metrics, and Challenges",Andrew Brown,3,Big Data and Cognitive Computing,57,"Background: Retrieval-augmented generation (RAG) aims to reduce hallucinations and outdated knowledge by grounding LLM outputs in retrieved evidence, but empirical results are scattered across tasks, systems, and metrics, limiting cumulative insight. Objective: We aimed to synthesise empirical evidence on RAG effectiveness versus parametric-only baselines, map datasets/architectures/evaluation practices, and surface limitations and research gaps. Methods: This systematic review was conducted and reported in accordance with PRISMA 2020. We searched the ACM Digital Library, IEEE Xplore, Scopus, ScienceDirect, and DBLP; all sources were last searched on 13 May 2025. This included studies from January 2020–May 2025 that addressed RAG or similar retrieval-supported systems producing text output, met citation thresholds (≥15 for 2025; ≥30 for 2024 or earlier), and offered original contributions; excluded non-English items, irrelevant works, duplicates, and records without accessible full text. Bias was appraised with a brief checklist; screening used one reviewer with an independent check and discussion. LLM suggestions were advisory only; 2025 citation thresholds were adjusted to limit citation-lag. We used a descriptive approach to synthesise the results, organising studies by themes aligned to RQ1–RQ4 and reporting summary counts/frequencies; no meta-analysis was undertaken due to heterogeneity of designs and metrics. Results: We included 128 studies spanning knowledge-intensive tasks (35/128; 27.3%), open-domain QA (20/128; 15.6%), software engineering (13/128; 10.2%), and medical domains (11/128; 8.6%). Methods have shifted from DPR + seq2seq baselines to modular, policy-driven RAG with hybrid/structure-aware retrieval, uncertainty-triggered loops, memory, and emerging multimodality. Evaluation remains overlap-heavy (EM/F1), with increasing use of retrieval diagnostics (e.g., Recall@k, MRR@k), human judgements, and LLM-as-judge protocols. Efficiency and security (poisoning, leakage, jailbreaks) are growing concerns. Discussion: Evidence supports a shift to modular, policy-driven RAG, combining hybrid/structure-aware retrieval, uncertainty-aware control, memory, and multimodality, to improve grounding and efficiency. To advance from prototypes to dependable systems, we recommend: (i) holistic benchmarks pairing quality with cost/latency and safety, (ii) budget-aware retrieval/tool-use policies, and (iii) provenance-aware pipelines that expose uncertainty and deliver traceable evidence. We note the evidence base may be affected by citation-lag from the inclusion thresholds and by English-only, five-library coverage. Funding: Advanced Research and Engineering Centre. Registration: Not registered." +W7155003632,10.48550/arxiv.2604.15385,10.48550/arxiv.2604.15385,2026,Prompt-Driven Code Summarization: A Systematic Literature Review,Afia Farjana,3,arXiv (Cornell University),0,"Software documentation is essential for program comprehension, developer onboarding, code review, and long-term maintenance. Yet producing quality documentation manually is time-consuming and frequently yields incomplete or inconsistent results. Large language models (LLMs) offer a promising solution by automatically generating natural language descriptions from source code, helping developers understand code more efficiently, facilitating maintenance, and supporting downstream activities such as defect localization and commit message generation. However, the effectiveness of LLMs in documentation tasks critically depends on how they are prompted. Properly structured instructions can substantially improve model performance, making prompt engineering-the design of input prompts to guide model behavior-a foundational technique in LLM-based software engineering. Approaches such as few-shot prompting, chain-of-thought reasoning, retrieval-augmented generation, and zero-shot learning show promise for code summarization, yet current research remains fragmented. There is limited understanding of which prompting strategies work best, for which models, and under what conditions. Moreover, evaluation practices vary widely, with most studies relying on overlap-based metrics that may not capture semantic quality. This systematic literature review consolidates existing evidence, categorizes prompting paradigms, examines their effectiveness, and identifies gaps to guide future research and practical adoption." +W7155245368,,,2026,Prompt-Driven Code Summarization: A Systematic Literature Review,Afia Farjana,3,arXiv (Cornell University),0,"Software documentation is essential for program comprehension, developer onboarding, code review, and long-term maintenance. Yet producing quality documentation manually is time-consuming and frequently yields incomplete or inconsistent results. Large language models (LLMs) offer a promising solution by automatically generating natural language descriptions from source code, helping developers understand code more efficiently, facilitating maintenance, and supporting downstream activities such as defect localization and commit message generation. However, the effectiveness of LLMs in documentation tasks critically depends on how they are prompted. Properly structured instructions can substantially improve model performance, making prompt engineering-the design of input prompts to guide model behavior-a foundational technique in LLM-based software engineering. Approaches such as few-shot prompting, chain-of-thought reasoning, retrieval-augmented generation, and zero-shot learning show promise for code summarization, yet current research remains fragmented. There is limited understanding of which prompting strategies work best, for which models, and under what conditions. Moreover, evaluation practices vary widely, with most studies relying on overlap-based metrics that may not capture semantic quality. This systematic literature review consolidates existing evidence, categorizes prompting paradigms, examines their effectiveness, and identifies gaps to guide future research and practical adoption." +W7112823045,,,2024,"Towards robust, secure, and privacy-aware large language models of code","YANG, Zhou",1,Singapore Management University Institutional Knowledge (InK) (Singapore Management University),0,"The field of software engineering has witnessed a surge in large language models specifically tailored to understand and process code, which we call large language models for code (LLM4Code). The increasing popularity of LLM4Code is inseparable from three key factors: the availability of extensive datasets compiled from diverse data sources, the advancements in deep learning algorithms and computational power that facilitate the training of these powerful models, and the active engagement and collaboration within the research community fostering innovation and the rapid exchange of ideas and methodologies. As evidenced by a series of studies, LLM4Code has been experiencing rapid development and achieving phenomenal success in various facets of software engineering. These powerful models have swiftly evolved from experimental prototypes to practical tools, integrating into the daily workflow of software developers around the globe. However, we unveil that LLM4Code fails to satisfy many non-functional requirements. For example, LLM4Code may not be robust, is vulnerable to data poisoning attacks, and may leak sensitive information. The dissertation starts with the first systematic literature review on the non-functional properties of LLM4Code, which provides a comprehensive understanding of the current status of LLM4Code. We identify six important non-functional properties of LLM4Code, including robustness, security, privacy, usability, explainability, and e ffi ciency. Then, this dissertation presents work on evaluating the robustness of LLM4Code. We highlight the naturalness requirements in operating adversarial attacks for LLM4Code. We propose using the masked language modeling capability of LLM4Code to generate adversarial perturbations that look natural to human developers. We design a two-step method to generate adversarial examples for code. Evaluation results show our method can generate more natural adversarial examples than existing methods, and our method can provide stronger robustness enhancement than baselines. The second work evaluates the security of LLM4Code, specifically the threats posed by backdoor attacks. In this work, we design a novel data poisoning method that can inject adaptive triggers into the training data of LLM4Code. The triggers are much stealthier than the baseline methods, capable of evading detection by both human developers and state-of-the-art defensive methods. Our evaluation results show that the stealthy backdoor attack can still achieve a high attack success rate. The third work unveils the memorization phenomena in LLM4Code. As part of the dissertation, we show that simple strategies can make models produce a vast amount of outputs that are exactly memorized from their training data. It is worrying that the memorized contents" +W4410205944,,10.1145/3696673.3723053,2025,The Role of Generative AI Models in Requirements Engineering: A Systematic Literature Review,Poonkuzhali Vasudevan,2,ACM Southeast Conference (ACMSE),9,"The software engineering field is experiencing rapid growth, driven by recent advancements in Artificial Intelligence (AI), particularly in Generative AI (GenAI) and Large Language Models (LLMs). Requirements Engineering (RE), a critical phase in software development, involves gathering and defining software requirements. However, research on the impact of GenAI and LLMs within RE remains limited. This paper examines the adoption of GenAI in RE, with the aim of exploring its practical implications, identifying current research trends, and highlighting areas for future development. To achieve this, a systematic literature review was conducted, addressing three research questions and analyzing 44 studies published over the past decade. The findings reveal that GenAI models, especially LLMs, are extensively employed in a variety of RE tasks, underscoring the versatility and potential of LLMs in enhancing the RE process." +W7160265065,,10.14569/ijacsa.2026.0170478,2026,The Impact of Modern AI on Software Development: A Systematic Literature Review,Meryem Bensaid,4,International Journal of Advanced Computer Science and Applications,0,"As large language models, and agentic AI systems are increasingly being integrated into software engineering, an expanding amount of empirical evidence surrounding these technologies has emerged. This systematic literature review examines the impact of modern AI techniques and tools in software development lifecycle phases and related activities, covering studies published between 2023 and 2025 and resulting in a corpus of 62 primary studies, investigating the role of large language models, AI agents and agentic AI workflows across lifecycle phases. The synthesis is guided by three research questions that address the entire software development cycle, reported impacts on development practices, outcomes, and constraints. This review fills a gap in the synthesis of modern AI applications and their impacts. It concludes that modern AI in software engineering is progressively evolving from a generation tool into a reasoning and coordination infrastructure layer, with ongoing efforts targeting the mitigation of identified limitations and the advancement of trustworthy agentic AI capabilities for dependable software engineering practices." +W4415939046,10.48550/arxiv.2511.00776,10.48550/arxiv.2511.00776,2025,"A Systematic Literature Review of Code Hallucinations in LLMs: Characterization, Mitigation Methods, Challenges, and Future Directions for Reliable AI",Cuiyun Gao,8,arXiv (Cornell University),2,"Model hallucination is one of the most critical challenges faced by Large Language Models (LLMs), especially in high-stakes code intelligence tasks. As LLMs become increasingly integrated into software engineering tasks, understanding and mitigating hallucination in code becomes essential. In this survey, we provide a systematic review of hallucination phenomena in code-oriented LLMs from four key perspectives. First, we begin by surveying 60 papers to define hallucination in the context of code and summarize its primary causes, such as data noise, exposure bias, and insufficient semantic grounding, while also tracing recent trends in literature across natural language processing (NLP) and software engineering communities. Second, we review model hallucination surveys in a broader span and summarize representative hallucination mitigation strategies, such as knowledge-enhanced generation, constrained decoding, and post-editing. Third, we review approaches targeted for code intelligence and highlight code-specific challenges that aggravate hallucination, including syntax sensitivity, strict type systems, and dependence on external libraries. Meanwhile, we analyze how emerging code intelligence tasks, e.g., program analysis, symbolic execution, and unit testing, are utilized to detect and mitigate hallucinations. Fourth, we summarize current evaluation benchmarks, ranging from static metrics to dynamic checks, e.g., compilation and execution correctness, and emphasize the need for hallucination-oriented benchmarks." +W7208824083,,10.66104/4hndmk68,2026,Artificial Intelligence Tools for Productivity Enhancement: A Systematic Literature Review,Marcelo Almeida Pereira Guedes,1,Nexus Science Review,1,"Artificial Intelligence (AI), particularly generative AI based on Large Language Models (LLMs), has rapidly transformed the execution of knowledge-intensive activities across multiple domains. AI-powered tools such as ChatGPT, GitHub Copilot, Microsoft Copilot, Google Gemini, Claude, and Notion AI have increasingly been adopted to automate repetitive tasks, support decision-making, accelerate software development, and improve content production. However, despite the rapid expansion of these technologies, scientific evidence regarding their effectiveness as productivity enhancers remains distributed across different research areas. This study presents a systematic literature review aimed at synthesizing current evidence on the role of AI tools in improving productivity in business, education, software engineering, and scientific research. A structured literature search was conducted across major academic databases, including Google Scholar, IEEE Xplore, ACM Digital Library, ScienceDirect, SpringerLink, and Scopus. Studies published between 2020 and 2026 were analyzed according to predefined inclusion and exclusion criteria. The reviewed literature indicates that AI tools can improve productivity by reducing task completion time, assisting knowledge creation, supporting programming activities, and enhancing information processing. Nevertheless, significant challenges remain, including inaccurate outputs, algorithmic bias, privacy concerns, ethical risks, and excessive reliance on automated systems. The findings suggest that AI achieves the greatest productivity benefits when used as a collaborative technology that augments human capabilities rather than replacing human expertise. Future research should investigate long-term productivity impacts, organizational adaptation strategies, responsible AI governance, and the integration of advanced multimodal AI systems into professional workflows." +W4414155484,,10.1145/3766552,2025,Metamorphic Testing of Deep Code Models: A Systematic Literature Review,Ali Asgari,4,ACM Transactions on Software Engineering and Methodology,3,"Large language models and deep learning models designed for code intelligence have revolutionized the software engineering field due to their ability to perform various code-related tasks. These models can process source code and software artifacts with high accuracy in tasks such as code completion, defect detection, and code summarization; therefore, they can potentially become an integral part of modern software engineering practices. Despite these capabilities, robustness remains a critical quality attribute for deep-code models as they may produce different results under varied and adversarial conditions (e.g., variable renaming). Metamorphic testing has become a widely used approach to evaluate models’ robustness by applying semantic-preserving transformations to input programs and analyzing the stability of model outputs. While prior research has explored testing deep learning models, this systematic literature review focuses specifically on metamorphic testing for deep code models. By studying 45 primary papers, we analyze the transformations, techniques, and evaluation methods used to assess robustness. Our review summarizes the current landscape, identifying frequently evaluated models, programming tasks, datasets, target languages, and evaluation metrics, and highlights key challenges and future directions for advancing the field." +W7126253154,10.48550/arxiv.2601.20879,10.48550/arxiv.2601.20879,2026,"A Survey on Large Language Model Impact on Software Evolvability and Maintainability: the Good, the Bad, the Ugly, and the Remedy",Bruno Claudino Matias,6,arXiv (Cornell University),0,"Context. Large Language Models (LLMs) are increasingly embedded in software engineering workflows for tasks including code generation, summarization, repair, and testing. Empirical studies report productivity gains, improved comprehension, and reduced cognitive load. However, evidence remains fragmented, and concerns persist about hallucinations, unstable outputs, methodological limitations, and emerging forms of technical debt. How these mixed effects shape long-term software maintainability and evolvability remains unclear. Objectives. This study systematically examines how LLMs influence the maintainability and evolvability of software systems. We identify which quality attributes are addressed in existing research, the positive impacts LLMs provide, the risks and weaknesses they introduce, and the mitigation strategies proposed in the literature. Method. We conducted a systematic literature review. Searches across ACM DL, IEEE Xplore, and Scopus (2020 to 2024) yielded 87 primary studies. Qualitative evidence was extracted through a calibrated multi-researcher process. Attributes were analyzed descriptively, while impacts, risks, weaknesses, and mitigation strategies were synthesized using a hybrid thematic approach supported by an LLM-assisted analysis tool with human-in-the-loop validation. Results. LLMs provide benefits such as improved analyzability, testability, code comprehension, debugging support, and automated repair. However, they also introduce risks, including hallucinated or incorrect outputs, brittleness to context, limited domain reasoning, unstable performance, and flaws in current evaluations, which threaten long-term evolvability. Conclusion. LLMs can strengthen maintainability and evolvability, but they also pose nontrivial risks to long-term sustainability. Responsible adoption requires safeguards, rigorous evaluation, and structured human oversight." +W7163154377,,10.22214/ijraset.2026.83359,2026,Generative AI in Mobile Application Development: A Systematic Literature Review,Balram Kumar,2,International Journal for Research in Applied Science and Engineering Technology,0,"Generative Artificial Intelligence (GenAI) tools have become central to modern software engineering, fundamentally transforming how developers design, write, debug, test, and document code. This systematic literature review examines the measurable impact of GenAI tools — specifically GitHub Copilot, OpenAI GPT-4/ChatGPT, Google Gemini, Anthropic Claude 3, and Meta Code LLaMA — on Android and Flutter mobile application development. Following the PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) framework, we screened 120 candidate papers and identified 35 high-quality peer-reviewed publications from January 2020 to March 2026 across five major academic databases. The review addresses four research questions covering tool usage patterns, productivity gains, quality and security risks, and future research directions. Findings confirm mean productivity gains of 45–55% for code generation, 30–40% for debugging, 35–50% for UI/UX automation, and up to 60% for documentation writing. However, 25–40% of AI-generated code contains correctness issues and approximately 40% of security-critical AI code contains exploitable vulnerabilities. A novel Seven-Phase GenAI Integration Framework is proposed, mapping specific tools to each phase of the mobile development lifecycle. Six priority areas for future research are identified, including mobile-specific fine-tuned models, real-time RAG for live API documentation, and on-device privacy-preserving LLMs." +W7165367028,,10.51582/interconf.19-20.05.2026.017,2026,Taxonomy of Prompt Injection Attacks and Analysis of Defense Mechanisms in Large Language Model-Based Chatbots,Telman Yusifov,3,InterConf,0,"Large Language Models have become critical infrastructure across healthcare, finance, and software engineering, yet their instruction-following design creates a structural class of security vulnerabilities known as prompt injection. This paper presents a comprehensive taxonomy of prompt injection attacks targeting LLM-based chatbots, organizes the threat landscape across seven categories covering both direct and indirect injection vectors, and evaluates defense mechanisms at three operational layers. The theoretical framework is validated through an empirical benchmark (PI-Bench) testing 210 prompt injection payloads across seven attack categories against five open-weight LLM models in a controlled local environment. Empirical findings demonstrate attack success rates ranging from 10.5% (LLaMA 3.1 8B) to 83.8% (Mistral 7B), confirming that alignment training quality is the primary determinant of injection resistance rather than model size. Across all models, Jailbreak/Roleplay attacks proved most effective at 34.7% aggregate ASR, while Gradient-Based attacks were least effective at 22.7%. The literature review documents attack success rates reaching 87.2% against state-of-the-art proprietary models and an 86.1% success rate for indirect injection against production applications. No single defensive measure provides adequate coverage, and a defense-in-depth architecture is necessary for robust protection." +W7171902466,,,2026,Preliminary Guidelines for Using and Evaluating GenAI Tools to Support Systematic Literature Reviews,Barbara Kitchenham,6,arXiv (Cornell University),0,"Context: Generative AI (GenAI) and Large Language Models (LLMs) are increasingly used for academic tasks in software engineering and beyond, including systematic literature reviews (SLRs). However, while capable of summarizing text, there is no guarantee they can meet the rigour, reliability, and transparency that SLRs require. Objectives: To support researchers intending to conduct SLRs using GenAI or those conducting empirical studies evaluating how well GenAI supports SLR tasks. Methods: First, we conducted a rapid review to identify studies that propose guidelines for evaluating and using GenAI and LLMs to support SLRs. Second, we drew on thought experiments, relevant guidance from the literature, and our own experience conducting SLRs and evaluating tools to develop recommendations for how to use and assess GenAI in the context of SLRs. Results: We discuss the problems researchers face when evaluating GenAI for SLRs. We identify and explain process issues to consider when planning, conducting, and reporting both SLRs using GenAI and evaluations of GenAI tools. Finally, we summarize our results as a set of process recommendations, which we name GUEST (GenAI Use and Evaluation in SLR Tasks). Conclusion: We argue that GenAI requires human oversight and is not currently capable of unsupervised systematic studies. However, it offers the prospect of cost-effective assistance for some repetitive tasks and for additional validation of some complex tasks. Our GUEST recommendations should help software engineering researchers both to conduct and report trustworthy SLRs using GenAI and to provide rigorous independent evaluation studies." +W7134063879,,10.17632/nf7dz42sfn,2026,Supplementary Material for A Conceptual Model for Born-Accessible Web Accessibility Remediation Using Large Language Models,Guillermo Vera-Amaro,2,Mendeley Data,0,"This dataset supports the transparency and reproducibility of the systematic literature review and procedures reported in the article ""A Conceptual Model for Born-Accessible Web Accessibility Remediation Using Large Language Models."" The folder ""01-Related Work search results"" documents the initial automated search process conducted in the Scopus and Web of Science databases. It includes the retrieved records based on the filters applied during the preliminary review of related literature. The folder ""02-Automatic search results"" contains the subsequent search stages carried out across selected digital library databases (Scopus, Web of Science, and arXiv). It includes exported bibliographic records organized by search strings, as well as the inclusion and exclusion steps applied. The file ""Dataset_Conceptual_model_Web_Accessibility.xlsx"" serves as the main dataset. It contains fully curated data from the entire review process, including metadata for the selected studies: titles, venues, publication year, AI techniques, LLMs used, prompting strategies, evaluation methods, and referenced accessibility standards. It also includes raw and normalized quality assessment scores, aligned with the Kitchenham's methodology for software engineering research. Also includes a link to the code repository for the validation phase of the model. Together, these materials enable full reproducibility of the review and provide a structured foundation for future research in AI-assisted web accessibility." +W7128496016,,10.1145/3796522,2026,A Systematic Literature Review of Parameter-Efficient Fine-Tuning for Large Code Models,Saima Afrin,3,ACM Transactions on Software Engineering and Methodology,1,"The rise of Artificial Intelligence (AI)-and particularly Large Language Models (LLMs) for code–has reshaped Software Engineering (SE) by enabling the automation of tasks such as code generation, bug detection, and repair. However, these models require significant computational resources for training and fine-tuning, posing challenges for real-world adoption in resource-constrained environments. To address this, the research community has increasingly turned to Parameter-Efficient Fine-Tuning (PEFT)–a class of techniques that enables the adaptation of large models by updating only a small subset of parameters, rather than the entire model. In this Systematic Literature Review (SLR), we examine the growing application of PEFT techniques–across a wide range of software engineering tasks. We analyze how these methods are used to optimize various deep learning (DL) architectures, focusing on their impact on both performance and efficiency. Our study synthesizes findings from 28 peer-reviewed papers, identifying patterns in configuration strategies and adaptation trade-offs. The outcome of this review is a comprehensive taxonomy that categorizes PEFT usage by task type, distinguishing between generative (e.g., Code Summarization) and non-generative (e.g., Code Clone Detection) scenarios. Our findings aim to inform future research and guide the practical deployment of PEFT in sustainable, AI-powered software development. Our artifacts are publicly available at https://github.com/alvi75/SLR-PEFT" +W7171690888,10.48550/arxiv.2607.24991,10.48550/arxiv.2607.24991,2026,Preliminary Guidelines for Using and Evaluating GenAI Tools to Support Systematic Literature Reviews,Barbara Kitchenham,6,arXiv (Cornell University),0,"Context: Generative AI (GenAI) and Large Language Models (LLMs) are increasingly used for academic tasks in software engineering and beyond, including systematic literature reviews (SLRs). However, while capable of summarizing text, there is no guarantee they can meet the rigour, reliability, and transparency that SLRs require. Objectives: To support researchers intending to conduct SLRs using GenAI or those conducting empirical studies evaluating how well GenAI supports SLR tasks. Methods: First, we conducted a rapid review to identify studies that propose guidelines for evaluating and using GenAI and LLMs to support SLRs. Second, we drew on thought experiments, relevant guidance from the literature, and our own experience conducting SLRs and evaluating tools to develop recommendations for how to use and assess GenAI in the context of SLRs. Results: We discuss the problems researchers face when evaluating GenAI for SLRs. We identify and explain process issues to consider when planning, conducting, and reporting both SLRs using GenAI and evaluations of GenAI tools. Finally, we summarize our results as a set of process recommendations, which we name GUEST (GenAI Use and Evaluation in SLR Tasks). Conclusion: We argue that GenAI requires human oversight and is not currently capable of unsupervised systematic studies. However, it offers the prospect of cost-effective assistance for some repetitive tasks and for additional validation of some complex tasks. Our GUEST recommendations should help software engineering researchers both to conduct and report trustworthy SLRs using GenAI and to provide rigorous independent evaluation studies." +W4417523743,10.48550/arxiv.2504.21569,10.48550/arxiv.2504.21569,2025,A Systematic Literature Review of Parameter-Efficient Fine-Tuning for Large Code Models,Md. Zahidul Haque,3,arXiv (Cornell University),1,"The rise of Artificial Intelligence (AI)-and particularly Large Language Models (LLMs) for code-has reshaped Software Engineering (SE) by enabling the automation of tasks such as code generation, bug detection, and repair. However, these models require significant computational resources for training and fine-tuning, posing challenges for real-world adoption in resource-constrained environments. To address this, the research community has increasingly turned to Parameter-Efficient Fine-Tuning (PEFT)-a class of techniques that enables the adaptation of large models by updating only a small subset of parameters, rather than the entire model. In this Systematic Literature Review (SLR), we examine the growing application of PEFT techniques-across a wide range of software engineering tasks. We analyze how these methods are used to optimize various deep learning (DL) architectures, focusing on their impact on both performance and efficiency. Our study synthesizes findings from 27 peer-reviewed papers, identifying patterns in configuration strategies and adaptation trade-offs. The outcome of this review is a comprehensive taxonomy that categorizes PEFT usage by task type, distinguishing between generative (e.g., Code Summarization) and non-generative (e.g., Code Clone Detection) scenarios. Our findings aim to inform future research and guide the practical deployment of PEFT in sustainable, AI-powered software development. Our artifacts are publicly available at https://github.com/alvi75/SLR-PEFT" +W4416642449,,10.1016/j.infsof.2025.107979,2025,Quality assessment of software requirements using artificial intelligence methods: A systematic literature review,Elise Wolf,3,Information and Software Technology,7,"Context: The quality of requirements specifications is a critical success factor in software development. Assuring high-quality requirements, specifically in an automated way, poses a significant challenge due to their unstructured and multi-modal character. With the rise of deep learning and large language models (LLMs), new opportunities have developed to assess the quality of requirements automatically, particularly user stories in the context of agile software engineering, where short development cycles require efficient tool support. Objective: This study aims to systematically review and investigate the current landscape of approaches based on artificial intelligence techniques such as natural language processing and deep learning for assessing the quality of software requirements. The investigation focuses on the artificial intelligence techniques adopted, quality aspects considered, datasets used to tune and evaluate the proposed approaches, and their performance. Method: We conducted a systematic literature review of 26 peer-reviewed papers published between 2019 and 2025. We selected the papers after a title and abstract review of 353 papers identified through a literature databases query and forward–backward snowballing. Results: The results reveal significant overlap among considered quality aspects, which can be mapped onto the higher-order requirements quality model INVEST. Most studies focus on assessing requirement quality rather than improving requirements and rely heavily on synthetic and public datasets. LLMs have rapidly gained popularity since 2023, though model evaluation strategies remain inconsistent. Metrics such as accuracy, precision, recall, and F1-Score are common, yet a few studies use semantic or expert-based evaluations. Conclusion: The field is evolving toward LLM-driven, semantically rich models, yet lacks methodological standardization, reproducible datasets for evaluating the models, and integration of the approaches with real-world requirements engineering processes. Future work should address these limitations by developing benchmark datasets, standardizing evaluation metrics, and exploring hybrid systems that combine AI-based and traditional requirements quality assurance approaches." +W4412081931,,10.1109/tse.2025.3586411,2025,"Open Source, Hidden Costs: A Systematic Literature Review on OSS License Management",Boyuan Li,6,IEEE Transactions on Software Engineering,5,"Integrating third-party software components is a common practice in modern software development, offering significant advantages in terms of efficiency and innovation. However, this practice is fraught with risks related to software licensing. A lack of understanding may lead to disputes, which can pose serious legal and operational challenges. To these ends, both academia and industry have conducted various investigations and proposed solutions and tools to deal with these challenges. However, significant limitations still remain. Moreover, the rapid evolution of open-source software (OSS) licenses, as well as the rapidly incorporated generative software engineering techniques, such as large language models for code (CodeLLMs), are placing greater demands on the systematic management of software license risks. To unveil the severe challenges and explore possible future directions, we conduct the first systematic literature review (SLR) on 80 carefully selected OSS license-related papers, classifying existing research into three key categories, i.e., license identification, license risk assessment, and license risk mitigation. Based on these, we discuss challenges in existing solutions, conclude the opportunities to shed light on future research directions and offer practical recommendations for practitioners.We hope this thorough review will help bridge the gaps between academia and industry and accelerate the ecosystem-wide governance of legitimate software risks within the software engineering community." +W7223205246,,10.17605/osf.io/cpzsa,2026,"Automated Adversary Emulation and Autonomous Penetration Testing: A Systematic Literature Review of Attack Graphs, Reinforcement Learning, LLM Agents, and Framework Validity",Lynet Tsitsi Svotwa,1,Open Science Framework,0,"This systematic literature review investigates the evolution of automated adversary emulation and autonomous penetration testing, from attack graphs and classical planning approaches to reinforcement learning and large language model (LLM) based autonomous agents. The review examines how these approaches model attacker objectives, represent target environments, select attack actions and evaluate performance. It also assesses empirical evidence concerning the validity and limitations of the Cyber Kill Chain and MITRE ATT&CK frameworks for representing adversary behaviour. The review will use structured academic searches, predefined eligibility criteria, transparent study selection, quality appraisal and comparative narrative synthesis. Reporting will be guided by PRISMA 2020 and PRISMA-S, with methodological consideration of established software-engineering systematic review guidance. Preliminary literature searches and paper retrieval have commenced. Formal screening, quality appraisal and synthesis are planned to follow the finalisation of this protocol" +W7167225127,,10.1016/j.infsof.2026.108246,2026,Architectural perspectives on GenAI-driven software testing systems: A systematic literature review,Mariam Lahami,2,Information and Software Technology,0, +W7162394858,10.48550/arxiv.2605.25536,10.48550/arxiv.2605.25536,2026,"A Tertiary Review of Large Language Model-Based Code Generating Tasks: Trends, Challenges, and Future Directions",Muslim Chochlov,3,arXiv (Cornell University),0,"Context. Large language models (LLMs) are increasingly applied to code-generating tasks (CGTs) in software engineering. While reported results are promising, the broader effects of such application and their integration into real-world development remain insufficiently understood with existing tertiary studies provide little in this area. Objective. This tertiary study consolidates secondary evidence on LLM-based CGTs, synthesizing the publication landscape, effects, scenarios, integration challenges, and future research directions. Method. Following systematic review guidelines, we searched in related digital libraries, complemented by backward-and-forward snowballing and screening step. Study quality was assessed and extraction reliability was audited with inter-rater agreement statistics. Evidence was synthesized using SWEBOK knowledge areas and the HELM framework. Results. We identify 30 secondary studies published between 2017-2025, with rapid growth since 2023. Accuracy seems strong on benchmarks but weakly supported for real-world generalization; robustness is fragile across tasks and configurations; efficiency constraints are pervasive; toxicity and bias are under-reported. Dominant challenges concern economic feasibility, evaluation validity, and socio-technical integration. Future directions suggest domain-aware model improvement and the need for holistic, standardized evaluation. Conclusion. LLM-based CGTs represent a fast-maturing yet unevenly evaluated research area, highlighting the need for domain-aware model improvements and holistic, standardized evaluation, addressing efficiency and associated costs." +W7220650326,,10.5281/zenodo.23187741,2026,Bibliographic References Repository - Triaxial Evaluation of Large Language Models in Code Generation: A Structural and Semantic Approach.,Kleber Fernando Dos Santos,1,Zenodo (CERN European Organization for Nuclear Research),0,"This repository provides the bibliographic dataset and methodological documentation supporting the research article *""Triaxial Evaluation of Large Language Models in Code Generation: A Structural and Semantic Approach""*. The dataset consists of a comprehensive BibTeX (`.bib`) file containing a hybrid corpus of scientific literature focused on the evaluation of AI-generated code, specifically regarding structural complexity, semantic correctness, and the application of Large Language Models (LLMs). The bibliographic collection was constructed through a dual-methodology approach: 1. **Automated Search (IEEE Xplore):** Executed between April 6 and April 20, 2026, targeting metadata fields (Abstract and Document Title) for key benchmarks such as ""HumanEval"", ""SWE-bench"", ""LiveCodeBench"", as well as queries intersecting LLMs, evaluation metrics, and coding.2. **Manual Curation:** Integration of state-of-the-art preprints (ArXiv) concerning LLM-as-a-judge evaluation paradigms, alongside seminal software engineering literature detailing classical complexity metrics (e.g., McCabe's Cyclomatic Complexity, Halstead's Software Science). Essential technical documentation and official repositories for the evaluated tools are also included. The provided `.bib` file represents the raw, unfiltered extraction prior to the application of the research's strict exclusion criteria, which subsequently isolated papers exclusively addressing metrics for AI-generated code. A `README.md` file is included in this repository, detailing the exact search strings, temporal scope, and the exclusion methodology required to reproduce the literature review process." +W7202261998,,10.5281/zenodo.21906011,2026,AI Agents in Support of Software Design and Architecture: Systematic Literature Review Protocol,RAMIRO DA CONCEIÇÃO DO NASCIMENTO JÚNIOR,11,Zenodo (CERN European Organization for Nuclear Research),0,"Protocol for a systematic literature review characterising the state ofthe art of LLM-based agents applied to software design and architecture,from two perspectives: the software engineering tasks addressed and theinternal architecture of the agents. The review follows Kitchenham andCharters (2007) with a hybrid search strategy and iterative snowballing(Wohlin, 2014), and is compatible with the PRISMA 2020 workflow. The search string was frozen on 10 July 2026 after achieving 100%sensitivity against a quasi-gold standard of 15 studies. This record is deposited before the screening phase begins. Subsequentamendments are published as new versions of this same record, preservingthe complete version history." +W4416283130,10.48550/arxiv.2507.05270,10.48550/arxiv.2507.05270,2025,"Open Source, Hidden Costs: A Systematic Literature Review on OSS License Management","Li, Boyuan",6,arXiv (Cornell University),0,"Integrating third-party software components is a common practice in modern software development, offering significant advantages in terms of efficiency and innovation. However, this practice is fraught with risks related to software licensing. A lack of understanding may lead to disputes, which can pose serious legal and operational challenges. To these ends, both academia and industry have conducted various investigations and proposed solutions and tools to deal with these challenges. However, significant limitations still remain. Moreover, the rapid evolution of open-source software (OSS) licenses, as well as the rapidly incorporated generative software engineering techniques, such as large language models for code (CodeLLMs), are placing greater demands on the systematic management of software license risks. To unveil the severe challenges and explore possible future directions, we conduct the first systematic literature review (SLR) on 80 carefully selected OSS license-related papers, classifying existing research into three key categories, i.e., license identification, license risk assessment, and license risk mitigation. Based on these, we discuss challenges in existing solutions, conclude the opportunities to shed light on future research directions and offer practical recommendations for practitioners. We hope this thorough review will help bridge the gaps between academia and industry and accelerate the ecosystem-wide governance of legitimate software risks within the software engineering community." +W7167467797,,10.59934/jaiea.v5i3.2431,2026,The Role of Generative AI in the Software Development Life Cycle: A Systematic Literature Review,Sebastian Saut Marulitua Sinaga,4,Journal of Artificial Intelligence and Engineering Applications (JAIEA),0,"The rapid advancement of Generative Artificial Intelligence (Generative AI), particularly through the emergence of Large Language Models (LLMs), has significantly transformed modern software engineering practices. These technologies enable automation across various phases of the Software Development Life Cycle (SDLC), including system design, coding, testing, and software maintenance. Despite their potential to improve development efficiency, the widespread adoption of Generative AI also introduces critical concerns related to software security, code quality, and long-term maintainability. This study aims to analyze the opportunities, security risks, and mitigation strategies associated with the integration of Generative AI into the SDLC. A Systematic Literature Review (SLR) with a qualitative descriptive approach was conducted by examining 15 primary studies published between 2021 and 2026, retrieved from IEEE Xplore, ACM Digital Library, Scopus, Google Scholar, and Portal Garuda. The collected literature was analyzed using content analysis and thematic analysis to identify the impacts of Generative AI across different SDLC phases. The findings reveal that Generative AI significantly enhances developer productivity, achieving efficiency gains of approximately 35% during system design, 55% during coding, 45% during testing, and 40% during software maintenance. However, AI-generated code remains vulnerable to various security weaknesses, including SQL Injection, Cross-Site Scripting (XSS), and improper input validation. Furthermore, excessive reliance on AI-generated outputs may contribute to technical debt accumulation through code duplication and reduced refactoring activities. To address these challenges, this study recommends the implementation of a Stratified AI-Human Governance Framework (SAHGF), which combines automated security validation, human code review, security testing, and continuous monitoring." +W7125596406,,10.1145/3787202,2026,Survey on Learning-based Dynamic Fault Localization: From Traditional Machine Learning to Large Language Models,Chunyan Liu,6,ACM Computing Surveys,4,"Learning-based dynamic fault localization techniques play a crucial role in the field of software engineering. These techniques dynamically execute test cases to meticulously extract useful knowledge from the execution information in the program, with the aim of identifying fault locations by leveraging machine learning, deep learning, and large language models. Currently, there is already a flourishing body of research that is intensely focused on learning-based dynamic fault localization. Research literature can be categorized into two main aspects for learning-based dynamic fault localization: data-based enhancements (i.e., the datasets) and model-based enhancements (i.e., the suspiciousness algorithms). Thus, we conduct an extensive literature review on learning-based dynamic fault localization from the aspects of the data task and the model task. Among them, each task is divided into multiple sub-tasks in a systematic manner to comprehensively discuss the details. In addition, we analyze and summarize the datasets and metrics that have been widely used to evaluate the effectiveness of the proposed techniques in recent years, so that researchers can have an intuitive perception of them. We also discuss the present challenges and the directions for future research." +W7133857781,,10.24406/publica-7754,2026,Quality assessment of software requirements using artificial intelligence methods: A systematic literature review,Elise Wolf,4,Fraunhofer-Publica (Fraunhofer-Gesellschaft),0,"Context: The quality of requirements specifications is a critical success factor in software development. Assuring high-quality requirements, specifically in an automated way, poses a significant challenge due to their unstructured and multi-modal character. With the rise of deep learning and large language models (LLMs), new opportunities have developed to assess the quality of requirements automatically, particularly user stories in the context of agile software engineering, where short development cycles require efficient tool support. Objective: This study aims to systematically review and investigate the current landscape of approaches based on artificial intelligence techniques such as natural language processing and deep learning for assessing the quality of software requirements. The investigation focuses on the artificial intelligence techniques adopted, quality aspects considered, datasets used to tune and evaluate the proposed approaches, and their performance. Method: We conducted a systematic literature review of 26 peer-reviewed papers published between 2019 and 2025. We selected the papers after a title and abstract review of 353 papers identified through a literature databases query and forward–backward snowballing. Results: The results reveal significant overlap among considered quality aspects, which can be mapped onto the higher-order requirements quality model INVEST. Most studies focus on assessing requirement quality rather than improving requirements and rely heavily on synthetic and public datasets. LLMs have rapidly gained popularity since 2023, though model evaluation strategies remain inconsistent. Metrics such as accuracy, precision, recall, and F1-Score are common, yet a few studies use semantic or expert-based evaluations. Conclusion: The field is evolving toward LLM-driven, semantically rich models, yet lacks methodological standardization, reproducible datasets for evaluating the models, and integration of the approaches with real-world requirements engineering processes. Future work should address these limitations by developing benchmark datasets, standardizing evaluation metrics, and exploring hybrid systems that combine AI-based and traditional requirements quality assurance approaches." +W7213930371,,10.31673/2409-7292.2026.035907,2026,METHOD OF CONTEXT-DRIVEN DYNAMIC SOFTWARE SECURITY TESTING USING LARGE LANGUAGE MODEL-BASED AGENTS,M. V. Maksimovich,1,Modern information security,0,"The advancement of artificial intelligence methods, in particular large language models and the agent-basedsystems built upon them, opens new opportunities for improving automated software security testing. Among the existingmethods, dynamic application security testing (DAST) is the most promising direction for such improvement, since itperforms end-to-end verification of a real running application and reflects its actual exploitable behaviour. At the sametime, classical DAST tools construct attack scenarios through stochastic enumeration based on predefined wordlists,which results in incomplete coverage and considerable resource consumption. The aim of this work is to develop a methodof context-driven dynamic software security testing based on agents built upon large language models, which increasesthe completeness of vulnerability detection and reduces the resource cost of scanning. The proposed method uses theOpenAPI specification as a source of semantic context, on the basis of which agents generate a set of end-to-end (E2E)security tests that act as a deterministic source of traffic and steer the dynamic scan through real business processes insteadof random crawling. The method provides two improvements: extended coverage of vulnerabilities that are unreachablefor signature-based scanners, and a reduced volume of redundant requests, since only the generated traffic is scanned. Avulnerability coverage model, the workflow, and the test scenario generation strategy are described, and metrics for evaluating the effectiveness of the method are defined. The results indicate the feasibility of the agent-based approach forextending coverage and improving the resource efficiency of dynamic testing, and outline the direction of furtherexperimental research.Keywords: dynamic application security testing, large language models, agent-based systems, OpenAPI, BOLA,IDOR, test automation. References1. Qadir, S., Waheed, E., Khanum, A., & Jehan, S. (2025). Comparative evaluation of approaches & tools foreffective security testing of Web applications. PeerJ Computer Science, 11, e2821. https://doi.org/10.7717/peerj-cs.2821.2. Xu, H., Wang, S., Li, N., Wang, K., Zhao, Y., Chen, K., Yu, T., Liu, Y., & Wang, H. (2025). Large languagemodels for cyber security: A systematic literature review. ACM Transactions on Software Engineering and Methodology.https://doi.org/10.1145/3769676.3. Eriksson, B., Pellegrino, G., & Sabelfeld, A. (2021). Black Widow: Blackbox data-driven web scanning. In2021 IEEE Symposium on Security and Privacy (SP) (pp. 1125–1142). IEEE. https://doi.org/10.1109/ SP40001.2021.00022.4. Xue, J., He, Y., Xiao, Y., Chen, Y., Wang, Z., & Lu, H. (2025). From heuristics to intelligence: Evolving blackbox scanners for modern web applications. In 2025 IEEE 10th International Conference on Data Science in Cyberspace(DSC) (pp. 247–254). IEEE. https://doi.org/10.1109/DSC67331.2025.00039.5. OWASP Foundation. (2023). API1:2023 Broken object level authorization. OWASP API Security Top 10 –2023. https://owasp.org/API-Security/editions/2023/en/0xa1-broken-object-level-authorization/.6. Alazmi, S., & De Leon, D. C. (2022). A systematic literature review on the characteristics and effectiveness ofweb application vulnerability scanners. IEEE Access, 10, 33200–33219. https://doi.org/10.1109/ACCESS.2022.3161522.7. Millar, S., Podgurskii, D., Kuykendall, D., Martínez del Rincón, J., & Miller, P. (2022). Optimising vulnerabilitytriage in DAST with deep learning. In Proceedings of the 15th ACM Workshop on Artificial Intelligence and Security(pp. 137–147). ACM. https://doi.org/10.1145/3560830.3563724.8. Khare, A., Dutta, S., Li, Z., Solko-Breslin, A., Alur, R., & Naik, M. (2025). Understanding the effectiveness oflarge language models in detecting security vulnerabilities. In 2025 IEEE Conference on Software Testing, Verificationand Validation (ICST) (pp. 103–114). IEEE. https://doi.org/10.1109/ICST62969.2025.10988968.9. Gnieciak, D., & Szandała, T. (2025). Large language models versus static code analysis tools: A systematicbenchmark for vulnerability detection. IEEE Access, 13, 198410–198422. https://doi.org/10.1109/ ACCESS.2025.3635168.10. Happe, A., & Cito, J. (2023). Getting pwn’d by AI: Penetration testing with large language models. InProceedings of the 31st ACM Joint European Software Engineering Conference and Symposium on the Foundations ofSoftware Engineering (pp. 2082–2086). ACM. https://doi.org/10.1145/3611643.3613083.11. Deng, G., Liu, Y., Mayoral-Vilches, V., Liu, P., Li, Y., Xu, Y., Zhang, T., Liu, Y., Pinzger, M., & Rass, S.(2024). PentestGPT: Evaluating and harnessing large language models for automated penetration testing. In Proceedingsof the 33rd USENIX Security Symposium (pp. 847–864). USENIX Association. https://www.usenix.org/conference/usenixsecurity24/presentation/deng.12. Shen, X., Wang, L., Li, Z., Chen, Y., Zhao, W., Sun, D., Wang, J., & Ruan, W. (2025). PentestAgent:Incorporating LLM agents to automated penetration testing. In Proceedings of the 20th ACM Asia Conference onComputer and Communications Security (pp. 375–391). ACM. https://doi.org/10.1145/3708821.3733882.13. Barabanov, A., Dergunov, D., Makrushin, D., & Teplov, A. (2022). Automatic detection of access controlvulnerabilities via API specification processing. arXiv. https://doi.org/10.48550/arXiv.2201.10833.14. Santos Filho, A., Rodríguez, R. J., & Feitosa, E. L. (2025). Automated broken object-level authorization attackdetection in REST APIs through OpenAPI to colored Petri nets transformation. International Journal of InformationSecurity, 24(2), 83. https://doi.org/10.1007/s10207-024-00970-5.15. Huang, Y., Shi, C., Lu, J., Li, H., Meng, H., & Li, L. (2024). Detecting broken object-level authorizationvulnerabilities in database-backed applications. In Proceedings of the 2024 ACM SIGSAC Conference on Computer andCommunications Security (pp. 2934–2948). ACM. https://doi.org/10.1145/3658644.3690227.16. Pasca, E. M., Delinschi, D., Erdei, R., & Matei, O. (2025). LLM-driven, self-improving framework for securitytest automation: Leveraging Karate DSL for augmented API resilience. IEEE Access, 13, 56861–56886.https://doi.org/10.1109/ACCESS.2025.3554960.17. Pasca, E. M., Erdei, R., Delinschi, D., & Matei, O. (2024). Enhancing API security testing against BOLA andauthentication vulnerabilities through an LLM-enhanced framework. In 19th International Conference on Soft ComputingModels in Industrial and Environmental Applications (SOCO 2024) (Lecture Notes in Networks and Systems, pp. 231–240). Springer. https://doi.org/10.1007/978-3-031-75010-6_23.18. OpenAPI Initiative. (2024). OpenAPI Specification v3.1.1. https://spec.openapis.org/oas/v3.1.1.html.19. ZAP (Zed Attack Proxy) [Програмне забезпечення]. (2025). Checkmarx. https://www.zaproxy.org/." +W7126280998,,,2026,Infusion of Blockchain to Establish Trustworthiness in AI Supported Software Evolution: A Systematic Literature Review,Mohammad Naserameri,2,arXiv (Cornell University),0,"Context: Blockchain and AI are increasingly explored to enhance trustworthiness in software engineering (SE), particularly in supporting software evolution tasks. Method: We conducted a systematic literature review (SLR) using a predefined protocol with clear eligibility criteria to ensure transparency, reproducibility, and minimized bias, synthesizing research on blockchain-enabled trust in AI-driven SE tools and processes. Results: Most studies focus on integrating AI in SE, with only 31% explicitly addressing trustworthiness. Our review highlights six recent studies exploring blockchain-based approaches to reinforce reliability, transparency, and accountability in AI-assisted SE tasks. Conclusion: Blockchain enhances trust by ensuring data immutability, model transparency, and lifecycle accountability, including federated learning with blockchain consensus and private data verification. However, inconsistent definitions of trust and limited real-world testing remain major challenges. Future work must develop measurable, reproducible trust frameworks to enable reliable, secure, and compliant AI-driven SE ecosystems, including applications involving large language models." +W7126195749,10.48550/arxiv.2601.20918,10.48550/arxiv.2601.20918,2026,Infusion of Blockchain to Establish Trustworthiness in AI Supported Software Evolution: A Systematic Literature Review,Mohammad Naserameri,2,arXiv (Cornell University),0,"Context: Blockchain and AI are increasingly explored to enhance trustworthiness in software engineering (SE), particularly in supporting software evolution tasks. Method: We conducted a systematic literature review (SLR) using a predefined protocol with clear eligibility criteria to ensure transparency, reproducibility, and minimized bias, synthesizing research on blockchain-enabled trust in AI-driven SE tools and processes. Results: Most studies focus on integrating AI in SE, with only 31% explicitly addressing trustworthiness. Our review highlights six recent studies exploring blockchain-based approaches to reinforce reliability, transparency, and accountability in AI-assisted SE tasks. Conclusion: Blockchain enhances trust by ensuring data immutability, model transparency, and lifecycle accountability, including federated learning with blockchain consensus and private data verification. However, inconsistent definitions of trust and limited real-world testing remain major challenges. Future work must develop measurable, reproducible trust frameworks to enable reliable, secure, and compliant AI-driven SE ecosystems, including applications involving large language models." +W7127972666,,10.70315/uloap.ulirs.2026.0301009,2026,Accelerating Mobile Application Development and Testing with Artificial Intelligence: A Systematic Literature Review,Maksym Yurko,1,Universal Library of Innovative Research and Studies,0,"The article presents a systematic literature review examining how artificial intelligence methods (LLM/GenAI, computer vision, deep learning, and multi-agent architectures) accelerate mobile application development and testing within the mobile SDLC. The objective is to address a deficit of domain-specific systematisation for mobile engineering and to answer three classes of questions: which AI approaches are applied across development and QA stages, which acceleration and efficiency metrics are empirically substantiated, and which quality/security risks accompany the adoption of generative tools. The relevance is driven by the growing complexity of mobile ecosystems and the limited scalability of manual testing and script-based automation, particularly due to brittleness under GUI changes. The novelty of the review lies in synthesising evidence from 28 selected studies from 2019 to 2025, with an explicit focus on mobile-specific constraints, and in juxtaposing speed gains against a trust contour. The principal findings are as follows: AI assistants demonstrate a substantial acceleration of routine tasks (up to ~55%) and a reduction in timelines for large-scale migrations. In testing, a transition toward LLM agents is observed, enabling high coverage at both scenario and element levels, as well as resilience to UI evolution. Concurrently, a trust crisis is documented due to a significant share of vulnerabilities in generated code, dependency hallucinations, and increased technical debt, necessitating the institutionalisation of verification practices and secure-by-default principles. The article is intended to be beneficial to software engineering researchers and to mobile development/QA practitioners integrating GenAI into workflows." +W7213309362,,10.5281/zenodo.22771852,2026,From Vibe Coding to Cognitive Augmentation: Rethinking the Role of Large Language Models in Software Development and Developer Learning,Glorious Babalola,1,Zenodo (CERN European Organization for Nuclear Research),0,"Overview This publication presents Version 1.0 of an ongoing research project examining the changing relationship between large language models (LLMs), AI-assisted programming, software development, programming education, and human problem-solving. The rapid adoption of generative AI coding tools has introduced a significant shift in how software is learned and produced. Developers and students can increasingly delegate activities such as code generation, debugging, explanation, documentation, and problem decomposition to AI systems. While existing research has identified substantial benefits in areas such as programming productivity, learning support, and reduction of cognitive workload, growing research also raises questions about over-reliance, cognitive offloading, critical thinking, verification, and the development of independent programming ability. This paper begins from that tension. Rather than asking whether AI is simply ""good"" or ""bad"" for programmers, the research explores a more fundamental question: Under what conditions can AI-assisted programming improve productivity and learning while preserving and developing the independent problem-solving abilities of software developers and programming learners? Purpose of Version 1.0 This first version is not presented as a completed empirical study. Instead, it establishes the literature and conceptual foundation for a broader research program. The paper reviews and synthesizes existing literature concerning: Large language models and AI-assisted programming Generative AI in programming education Developer productivity and AI coding assistants Cognitive offloading and human–AI interaction Critical thinking and generative AI AI-assisted learning versus AI-assisted production The experiences and practices of developers using AI coding tools Risks associated with excessive dependence on AI-generated solutions Based on this literature, the paper introduces a preliminary conceptual framework for cognitive-preserving AI use in software development and developer learning. Proposed Research Direction The paper proposes that the central challenge is not whether developers should use AI, but how AI should be integrated into the development and learning process. The proposed approach distinguishes between AI use that substitutes for human reasoning and AI use that augments it. The preliminary framework, referred to as the Cognitive-Preserving AI Development Framework (CPADF), emphasizes the following cycle: Think → Attempt → Ask → Evaluate → Implement → Explain → Reproduce The framework is intended to encourage developers and learners to retain meaningful engagement with problem formulation, reasoning, debugging, evaluation, explanation, and independent reproduction, while still benefiting from the productivity and informational capabilities of AI systems. The framework is conceptual at this stage and should therefore be regarded as a research proposition requiring further investigation, rather than as a validated intervention or established methodology. Position Within the Broader Research Project This paper represents the first contribution to an evolving research project. Future stages of the project may investigate the proposed framework through additional literature review, expert evaluation, surveys, interviews, controlled experiments, observational studies, and other empirical methods. Potential areas for future investigation include: The relationship between AI-assisted programming and independent debugging ability. Differences between unrestricted AI use and structured, cognitive-preserving AI use. The effects of AI assistance on programming learning and long-term retention. Differences in AI use between novice, intermediate, and experienced developers. The relationship between confidence in AI-generated code and critical evaluation. The effects of repeated AI dependence on opportunities to practice programming problem-solving. Whether structured AI-use frameworks can preserve learning while maintaining productivity benefits. How these effects may differ across educational and professional contexts. Future versions may incorporate additional evidence, expert feedback, empirical findings, methodological refinements, and collaboration with researchers, educators, and software engineering practitioners. Intended Contribution The objective of this work is not to argue that AI-assisted programming is inherently harmful, nor to reject the use of AI coding tools. Instead, it seeks to contribute to an emerging discussion about human–AI collaboration in software development by proposing a distinction between cognitive substitution and cognitive augmentation. The broader goal is to explore how developers and programming learners can use increasingly capable AI systems without unnecessarily delegating the very cognitive processes that they need to develop and maintain. This work is therefore intended as an open research contribution and starting point for discussion, criticism, replication, and future empirical investigation. Researchers, educators, software engineering practitioners, and other interested contributors are invited to review, challenge, extend, or collaborate on subsequent stages of the research. Research Status Version: 1.0Type: Literature Review and Conceptual Research PaperStatus: Open Working Paper / Initial Research ContributionEmpirical Study: Not conducted in this versionResearch Stage: Literature synthesis and conceptual framework development This version should not be interpreted as a peer-reviewed empirical study or as definitive evidence for the proposed framework. Its primary purpose is to establish the research problem, synthesize relevant literature, identify areas requiring further investigation, and provide a conceptual foundation for subsequent research. Collaboration The research project is intentionally being developed as an open and evolving initiative. Researchers and practitioners with expertise in areas such as: Artificial intelligence Large language models Software engineering Programming education Human–AI interaction Cognitive science Learning sciences Developer productivity Computing education are welcome to provide critical feedback, suggest relevant literature, contribute methodological perspectives, or explore potential collaboration for future stages of the project. Substantive contributions to future research will be appropriately recognized according to established academic authorship and contribution practices. Keywords: Large Language Models; Generative AI; AI-Assisted Programming; Software Engineering; Programming Education; Developer Productivity; Cognitive Offloading; Critical Thinking; Human–AI Interaction; AI in Education; Vibe Coding; Cognitive Augmentation; Cognitive Substitution; AI Coding Assistants" +W7160995210,,10.1007/s10515-026-00629-6,2026,An exploratory study on effectiveness of GPT-4o in conducting sub-tasks of systematic literature reviews,Mohammad Shameem,6,Automated Software Engineering,2, +W7165574043,,10.1016/j.infsof.2026.108232,2026,AI-driven self-healing across the edge–cloud continuum: A systematic literature review,Lander Bonilla,4,Information and Software Technology,2,"Context: The increasing complexity posed by Edge–Cloud Continuum architectures requires innovative software development practices and paradigms. Traditional software engineering practices fall short for addressing the particularities of such deployments; hence, there is a critical need for innovative tools and techniques capable of autonomous self-remediation and enhanced resilience with minimal human intervention. Objective: The overarching goal of this study is to synthesize the state-of-the-art with regard to AI-driven self-healing in the computing continuum. Implementation challenges and ethical implications are analyzed to identify effective approaches, architectures, and future trends in the field of automated program repair. Methods: A Systematic Literature Review was conducted following the PRISMA 2020 guidelines and Kitchenham’s methodology for software engineering. A rigorous automated search across designated academic databases initially yielded 2930 records. After applying strict inclusion and exclusion criteria, 99 primary studies were selected. Results: The analysis reveals a fundamental paradigm shift from observational monitoring to actionable, autonomous repair. While LLM-driven approaches and Agentic frameworks demonstrate high efficacy in tasks such as causal discovery and complex code patching, a critical validation gap remains regarding their deployment in non-deterministic runtime environments. Current literature often lacks the rigorous testing frameworks required to ensure that AI-generated repairs do not introduce regressions or security vulnerabilities. Conclusion: Achieving trustworthy autonomy requires a holistic approach that merges advanced AI paradigms with disciplined software engineering. Future research must prioritize the creation of standardized benchmarks and empirical studies to bridge the gap between theoretical algorithms and industrial application. Furthermore, the development of decentralized multi-agent architectures and Human-in-the-Loop techniques is essential to ensure safety, explainability, and ethical compliance in the next generation of resilient software systems." +W7200674033,,,2025,Experiences with integrating AI tools into software engineering education: A literature review,Ilari Matikainen,1,Työväentutkimus Vuosikirja,0,"Due to landmark developments in large language model technologies during recent years, a large influx of novel artificial intelligence software has created newfound interest in utilizing AI tools for a variety of different use cases. This thesis looks into the early experiences of integrating AI tools into software engineering education, and examines the effects that their use has had on learning and teaching software engineering. The study is conducted through a literature review. The data used for the literature review is a collection of previous studies on the use of AI tools in the classroom, with focus on their practical applications. The main findings are that both ChatGPT and custom-developed educational AI tools are being utilized actively, and that their use has generally had a positive effect on multiple aspects of software engineering education. Negative effects were identified too, however. These findings suggest that artificial intelligence will most likely have a larger role in the future education of software engineers, and different ways of utilizing AI tools in educational interventions should be further researched." diff --git a/02-literature/search/results/S1-openalex-qknorm.csv b/02-literature/search/results/S1-openalex-qknorm.csv new file mode 100644 index 0000000..0ad6db4 --- /dev/null +++ b/02-literature/search/results/S1-openalex-qknorm.csv @@ -0,0 +1,47 @@ +source_id,arxiv,doi,year,title,first_author,n_authors,venue,citations,abstract +W3099096615,,10.18653/v1/2020.findings-emnlp.379,2020,Query-Key Normalization for Transformers,Alex Henry,4,Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP),45,"Low-resource language translation is a challenging but socially valuable NLP task.Building on recent work adapting the Transformer's normalization to this setting, we propose QKNORM, a normalization technique that modifies the attention mechanism to make the softmax function less prone to arbitrary saturation without sacrificing expressivity.Specifically, we apply ℓ 2 normalization along the head dimension of each query and key matrix prior to multiplying them and then scale up by a learnable parameter instead of dividing by the square root of the embedding dimension.We show improvements averaging 0.928 BLEU over state-of-the-art bilingual benchmarks for 5 low-resource translation pairs from the TED Talks corpus and IWSLT'15. 1" +W7164894731,10.48550/arxiv.2606.16310,10.48550/arxiv.2606.16310,2026,QK-Normed MLA: QK normalization without full key caching,Yizhou Han,5,arXiv (Cornell University),0,"Query-key (QK) normalization stabilizes attention by controlling the scale of queries and keys before the dot product, but is not immediately compatible with Multi-head Latent Attention (MLA). MLA achieves efficient decoding by caching low-dimensional latent states instead of full keys, whereas post-projection QK RMSNorm appears to require the fully projected key for every cached token. We show this apparent incompatibility is an implementation artifact, not an architectural constraint. RMSNorm decomposes into a static affine weight and a dynamic scalar RMS statistic. The static key-side weight can be absorbed into the MLA query-side projection; the dynamic key statistic reduces to one inverse-RMS scalar per token and KV group. The resulting formulation is exactly equivalent to explicit post-projection QK RMSNorm in exact arithmetic and preserves MLA's latent decode path. In our 400M runs trained for up to 100B tokens, QK-Normed MLA achieves lower training loss and better downstream accuracy than QK clipping, while H800 decode benchmarks show less than 2% latency overhead up to 256k context. These results make QK normalization a practical stabilization option for MLA models without requiring full-key caching." +W7197026758,,10.1109/lca.2026.3720688,2026,SwiftQK: Fast and Communication-Efficient Tensor Parallelism for Query-Key Normalization,Gyudong Kim,3,IEEE Computer Architecture Letters,0,"Query-Key Normalization (QK-Norm) improves the training stability and quality of modern Large Language Models (LLMs). However, under Tensor Parallelism (TP), layerwise QK-Norm introduces additional cross-GPU communication because the normalization factor depends on the full hidden vector. We present SwiftQK, a multi-GPU RMSNorm kernel that exchanges only scalar normalization statistics and overlaps the remaining Peer-to-Peer reduction with independent element-wise computation in a deadlock-safe persistent kernel. Evaluations on recent LLMs show that SwiftQK reduces QK-Norm latency by 81.4--93.9% relative to the standard TP QK-Norm using full-vector All-Gather. In end-to-end serving, SwiftQK reduces TPOT on average by 29.5% over the All-Gather-based baseline and by 14.3% over an optimized scalar-aggregation implementation." +W4414997173,,10.20533/ijicr.2042.4655.2025.0165,2025,Attention Mechanism Enhancements in Modern Machine Learning,B. Thomas,1,International Journal of Intelligent Computing Research,0,"Attention mechanisms have become a cornerstone of modern machine learning, enabling models (especially Transformers) to weigh the importance of different input elements dynamically.Since the introduction of the Transformer and its Multi-Head Self-Attention (MHA) in 2017, attention-based models have achieved state-of-the-art results in NLP, computer vision, and other domains.However, the flexibility of attention comes at a high computational and memory cost: naïve self-attention requires O(n²) operations and memory with respect to sequence length n.This poses challenges for long sequences and efficient inference, motivating a surge of research into faster and more memory-efficient attention mechanisms.In this article, we review the fundamentals of attention and highlight recent advances aimed at improving attention speed and efficiency, including Multi-Query and Grouped-Query Attention, Rotary Positional Embeddings (RoPE), Query-Key Normalization (QK Norm), hybrid Transformer models, and high-performance implementations like FlashAttention." +W4416789464,10.48550/arxiv.2511.21377,10.48550/arxiv.2511.21377,2025,Controlling changes to attention logits,Ben Anson,2,arXiv (Cornell University),0,"Stability of neural network weights is critical when training transformer models. The query and key weights are particularly problematic, as they tend to grow large without any intervention. Applying normalization to queries and keys, known as `QK norm', fixes stability issues in practice, but is not always applicable. For example, QK norm is not compatible with Multi Latent Attention (MLA) because QK norm requires full materialization of queries and keys during inference, which is not done in MLA. In this paper we suggest that controlling the changes to logits is important for stability. We show that these changes are controllable by assigning parameter-dependent learning rates to the query and key weights. We find that our cheap intervention allows us to increase the base learning rate of the network, outperform other methods in the MLA setting, and achieve performance competitive with QK norm when using Multi-head Attention." +W4407092995,,10.52202/085713-2150,2025,Rope to Nope and Back Again: A New Hybrid Attention Strategy,Bowen Yang,7,neural information processing systems,0,"Long-context large language models (LLMs) have achieved remarkable advancements, driven by techniques like Rotary Position Embedding (RoPE) (Su et al., 2023) and its extensions (Chen et al., 2023; Liu et al., 2024c; Peng et al., 2023). By adjusting RoPE parameters and incorporating training data with extended contexts, we can train performant models with considerably longer input sequences. However, existing RoPE-based methods exhibit performance limitations when applied to extended context lengths. This paper presents a comprehensive analysis of various attention mechanisms, including RoPE, No Positional Embedding (NoPE), and Query-Key Normalization (QK-Norm), identifying their strengths and shortcomings in long-context modeling. Our investigation identifies distinctive attention patterns in these methods and highlights their impact on long-context performance, providing valuable insights for architectural design. Building on these findings, we propose a novel architecture featuring a hybrid attention mechanism that integrates global and local attention spans. This design not only surpasses conventional RoPE-based transformer models with full attention in both long and short context tasks but also delivers substantial efficiency gains during training and inference." +W4395483230,10.48550/arxiv.2404.15702,10.48550/arxiv.2404.15702,2024,Nyonic Technical Report,Junfeng Tian,6,arXiv (Cornell University),0,"This report details the development and key achievements of our latest language model designed for custom large language models. The advancements introduced include a novel Online Data Scheduler that supports flexible training data adjustments and curriculum learning. The model's architecture is fortified with state-of-the-art techniques such as Rotary Positional Embeddings, QK-LayerNorm, and a specially crafted multilingual tokenizer to enhance stability and performance. Moreover, our robust training framework incorporates advanced monitoring and rapid recovery features to ensure optimal efficiency. Our Wonton 7B model has demonstrated competitive performance on a range of multilingual and English benchmarks. Future developments will prioritize narrowing the performance gap with more extensively trained models, thereby enhancing the model's real-world efficacy and adaptability.GitHub: \url{https://github.com/nyonicai/nyonic-public}" +W7165558066,,10.5281/zenodo.20803082,2026,Gating and Normalization Interact in the Attention Block,Shubham Kumar,1,Zenodo (CERN European Organization for Nuclear Research),0,"Abstract We study how an elementwise sigmoid attention-output gate interacts with normalization placement in small causal transformers (29–36M parameters). A 54-model main sweep crosses three schemes (pre-norm; pre-norm + QK-norm; sandwich + QK-norm), three depths, three seeds, and models with and without the gate on WikiText-2. At the best validation checkpoint selected for each run, gating lowers mean validation perplexity in all nine normalization-depth cells. Averaged across depths and seeds, the gated models improve over their matched no-gate baselines by 1.03 PPL with pre-norm, 2.64 with pre-norm + QK-norm, and 3.35 with sandwich + QK-norm. The improvement is therefore 2.32 PPL larger for sandwich + QK-norm than for pre-norm (p < 0.001). A separate fixed-budget 8-layer experiment on WikiText-103 independently finds a gate benefit under all three schemes (all p < 0.01). Normalization placement also produces qualitatively different learned gate distributions: C1 and C2 become low-mean suppressors with essentially no fully-open values, while C3 becomes bimodal, with roughly 20% of values fully closed and 10% fully open. A single-layer ablation identifies the attention-output RMSNorm as necessary and sufficient for this bimodality at similar perplexity, and the distinction persists across the stable learning-rate range. Gate sparsity consistently peaks in the second transformer block. These results show that normalization placement controls both the performance effect and learned geometry of attention-output gating at this scale. Version 2 This version substantially expands the original manuscript with: a fixed-budget 8-layer experiment on WikiText-103; a single-layer ablation isolating the role of the attention-output RMSNorm; learning-rate robustness analyses; a parameter-matched no-gate control; expanded analysis of gate distributions and layer-wise sparsity; additional discussion, limitations, tables, and supplementary results. The experiments use small causal transformers trained from scratch and characterize behaviour in a compute-limited regime. The results should not be assumed to generalise to fully converged or large-scale language-model training without further evidence." +W4414271954,,10.1109/iceict66683.2025.11159965,2025,Enhanced Monocular Depth Estimation Based on Improved Self-Attention Mechanisms and Composite Loss Functions,Mingshen Huang,5,IEEE International Conference on Electronic Information and Communication Technology (ICEICT),0,"Monocular depth estimation, as a key technology in computer vision, has been widely applied in autonomous driving, robotic navigation, and other fields. However, balancing the accuracy of global depth prediction with the fine recovery of local details remains challenging. This paper presents an improved monocular depth estimation framework based on the powerful Depth Anything V2 pre-trained model. First, we fine-tune the ViT-Large pre-trained weights and integrate Query-Key Normalization (QK-Norm), Relative Position Bias (RelPosBias), and Rotary Position Embedding (RoPE) mechanisms into its core self-attention module. These mechanisms enhance the model's ability to perceive spatial information and relative positional relationships, improving the stability and position sensitivity of feature representation. Second, to address common issues of local discontinuities and edge blurring in depth prediction, we construct a composite loss function consisting of improved Scale-invariant Logarithmic Loss (SiLogLoss), Gradient Consistency Loss (GradientLoss), and Edge Smoothness Loss (EdgeSmoothnessLoss), providing supervision from global error, depth map structure, and edge details. We conducted comprehensive experimental evaluations on the KITTI dataset. Results demonstrate that our method achieves significant improvements over the original Depth Anything V2 model in key performance metrics: reducing the Absolute Relative Error (AbsRel) from 0.074 to 0.064, and increasing the $\delta_1$ accuracy from 0.946 to 0.967, validating the effectiveness of our proposed strategies." +W7172429293,10.48550/arxiv.2608.02091,10.48550/arxiv.2608.02091,2026,"One QK Channel, Many Sources: Tracing Low-Precision Attention Collapse",Shuxiao Xie,6,arXiv (Cornell University),0,"A bfloat16 transformer can train normally, then collapse abruptly. Prior work links collapse to structured attention errors and shows QK normalization disrupts their compounding. Distinct low-precision errors trigger the same collapse, leaving unclear whether each needs a fix at its source or one shared route can be blocked instead. We isolated the fault behind a reproduced GPT-2-class collapse to the streaming-softmax accumulator, where an fp32 streaming core repairs it, and turned it into an assay for moving a controlled error across sources. Using it, we found that errors placed outside attention still drove the same QK spectral runaway, and that correcting only QK kept training stable while the fault stayed active. This is a source-channel dissociation: fault source is not failure channel. It held across tested architectures and scales, and reproduced on a second GPU architecture. As a causal probe, projecting each update off the current QK weights' leading three singular directions held the query projection's largest singular value to 11.1, whereas removing equal energy elsewhere left it at 237: the QK channel causally drives the early runaway. What lets the injected error in is temporal sign-coherence, its per-head sign persisting across steps, not aggregate deviation; once inside, the runaway shows as attention-logit saturation. QK-Guard, a dormant controller, tests this by switching on parameter-free QK normalization at the first monitored threshold crossing. On the runs designated for this test, the QK-local action prevented the failure of each matched or same-configuration unguarded run; on plain GPT-2, all 12 final train and validation losses were within 0.03 nat of same-configuration always-on QK-norm, and both methods ran 60k steps without collapse. Intervention at the QK locus therefore suffices in place of a fix at each source." +W7127631375,10.48550/arxiv.2602.02522,10.48550/arxiv.2602.02522,2026,IMU-1: Sample-Efficient Pre-training of Small Language Models,George K. Grigorev,1,arXiv (Cornell University),0,"We present IMU-1, a 430M-parameter language model trained on 72B tokens that approaches the benchmark performance of models trained on 56x more data. We describe a validated training recipe combining recent architectural interventions (QK-norm attention, per-head gating, value residuals, LayerNorm scaling) with optimization advances (NorMuon with cautious weight decay, muP parametrization) and a three-stage training schedule with post-hoc checkpoint EMA. We provide ablations for each component and release code, weights and data to enable reproduction: https://huggingface.co/thepowerfuldeez/imu1_base" +W7214536204,,10.5281/zenodo.23001605,2026,Experiment Receipt Discipline for Small-Model Autoresearch,Reuben Bowlby,1,Zenodo (CERN European Organization for Nuclear Research),0,"Small-model autoresearch loops can generate dozens of experiments per session, but without receipt discipline the results become unauditable: hypotheses blur into outcomes, baselines drift, noise floors go unmeasured, and negative results disappear. This paper presents a case study of the autoresearch-pipeline corpus — 243 result rows, 55 kept patches, 186 DISCARD negative results, 2 crashes — as a worked example of receipt-bearing autoresearch. The strongest finding is not a new architecture patch; it is the governance value of a prospective noise-floor gate that blocks architecture experiments before the noise floor is measured, and the preservation of negative results as first-class evidence. We present the QK-Norm noise-floor-gate vignette as the centerpiece: a case where the noise floor (0.180 untuned) was 36–90× larger than the expected improvement (0.002–0.005 bpb), which the receipt discipline caught before any architecture experiment was authorized." +W4417119521,,10.1109/icassp55912.2026.11463059,2026,DiTSinger: Scaling Singing Voice Synthesis with Diffusion Transformer and Implicit Alignment,Zongcai Du,11,IEEE International Conference on Acoustics Speech and Signal Processing,0,"Recent progress in diffusion-based Singing Voice Synthesis (SVS) demonstrates strong expressiveness but remains limited by data scarcity and model scalability. We introduce a two-stage pipeline: a compact seed set of human-sung recordings is constructed by pairing fixed melodies with diverse LLM-generated lyrics, and melody-specific models are trained to synthesize over 500 hours of high-quality Chinese singing data. Building on this corpus, we propose DiTSinger, a Diffusion Transformer with RoPE and qk-norm, systematically scaled in depth, width, and resolution for enhanced fidelity. Furthermore, we design an implicit alignment mechanism that obviates phoneme-level duration labels by constraining phoneme-to-acoustic attention within character-level spans, thereby improving robustness under noisy or uncertain alignments. Extensive experiments validate that our approach enables scalable, alignment-free, and high-fidelity SVS." +W7135176124,10.48550/arxiv.2603.11611,10.48550/arxiv.2603.11611,2026,"Fractional Rotation, Full Potential? Investigating Performance and Convergence of Partial RoPE",Mohammad Aflah Khan,4,arXiv (Cornell University),0,"Rotary Positional Embedding (RoPE) is widely used in transformers to encode relative positional information, yet the effect of applying RoPE to only a subset of hidden dimensions remains underexplored. This design choice can substantially reduce memory usage, especially at long context lengths. We show that partial RoPE achieves up to 10$\times$ memory savings over standard RoPE cache while maintaining comparable final loss. We systematically study its impact on training dynamics and convergence across architectures, model sizes, sequence lengths, and datasets. Our results reveal that: (1) applying RoPE to only a small fraction of dimensions (around 10\%) matches the convergence of full RoPE; (2) these trends remain consistent across settings; and (3) NoPE (No Positional Encoding) can produce loss spikes, which are mitigated by minimal RoPE usage or QK-Norm application. Overall, our findings highlight partial RoPE as an effective trade-off between efficiency and stability." +W7133339030,10.48550/arxiv.2603.02170,10.48550/arxiv.2603.02170,2026,SageBwd: A Trainable Low-bit Attention,Jintao Zhang,8,arXiv (Cornell University),0,"Low-bit attention, such as SageAttention, has emerged as an effective approach for accelerating model inference, but its applicability to training remains poorly understood. In prior work, we introduced SageBwd, a trainable INT8 attention that quantizes six of seven attention matrix multiplications while preserving fine-tuning performance. However, SageBwd exhibited a persistent performance gap to full-precision attention (FPA) during pre-training. In this work, we investigate why this gap occurs and demonstrate that SageBwd matches full-precision attention during pretraining. Through experiments and theoretical analysis, we reach a few important insights and conclusions: (i) QK-norm is necessary for stable training at large tokens per step, (ii) quantization errors primarily arise from the backward-pass score gradient dS, (iii) reducing tokens per step enables SageBwd to match FPA performance in pre-training, and (iv) K-smoothing remains essential for training stability, while Q-smoothing provides limited benefit during pre-training." +W7128556445,,,2026,OneLive: Dynamically Unified Generative Framework for Live-Streaming Recommendation,Shen Wang,18,arXiv (Cornell University),0,"Live-streaming recommender system serves as critical infrastructure that bridges the patterns of real-time interactions between users and authors. Similar to traditional industrial recommender systems, live-streaming recommendation also relies on cascade architectures to support large-scale concurrency. Recent advances in generative recommendation unify the multi-stage recommendation process with Transformer-based architectures, offering improved scalability and higher computational efficiency. However, the inherent complexity of live-streaming prevents the direct transfer of these methods to live-streaming scenario, where continuously evolving content, limited lifecycles, strict real-time constraints, and heterogeneous multi-objectives introduce unique challenges that invalidate static tokenization and conventional model framework. To address these issues, we propose OneLive, a dynamically unified generative recommendation framework tailored for live-streaming scenario. OneLive integrates four key components: (i) A Dynamic Tokenizer that continuously encodes evolving real-time live content fused with behavior signal through residual quantization; (ii) A Time-Aware Gated Attention mechanism that explicitly models temporal dynamics for timely decision making; (iii) An efficient decoder-only generative architecture enhanced with Sequential MTP and QK Norm for stable training and accelerated inference; (iv) A Unified Multi-Objective Alignment Framework reinforces policy optimization for personalized preferences." +W7128476908,10.48550/arxiv.2602.08612,10.48550/arxiv.2602.08612,2026,OneLive: Dynamically Unified Generative Framework for Live-Streaming Recommendation,Shen Wang,18,arXiv (Cornell University),0,"Live-streaming recommender system serves as critical infrastructure that bridges the patterns of real-time interactions between users and authors. Similar to traditional industrial recommender systems, live-streaming recommendation also relies on cascade architectures to support large-scale concurrency. Recent advances in generative recommendation unify the multi-stage recommendation process with Transformer-based architectures, offering improved scalability and higher computational efficiency. However, the inherent complexity of live-streaming prevents the direct transfer of these methods to live-streaming scenario, where continuously evolving content, limited lifecycles, strict real-time constraints, and heterogeneous multi-objectives introduce unique challenges that invalidate static tokenization and conventional model framework. To address these issues, we propose OneLive, a dynamically unified generative recommendation framework tailored for live-streaming scenario. OneLive integrates four key components: (i) A Dynamic Tokenizer that continuously encodes evolving real-time live content fused with behavior signal through residual quantization; (ii) A Time-Aware Gated Attention mechanism that explicitly models temporal dynamics for timely decision making; (iii) An efficient decoder-only generative architecture enhanced with Sequential MTP and QK Norm for stable training and accelerated inference; (iv) A Unified Multi-Objective Alignment Framework reinforces policy optimization for personalized preferences." +W4415310639,10.48550/arxiv.2507.13338,10.48550/arxiv.2507.13338,2025,Training Transformers with Enforced Lipschitz Constants,Laker Newhouse,6,arXiv (Cornell University),1,"Neural networks are often highly sensitive to input and weight perturbations. This sensitivity has been linked to pathologies such as vulnerability to adversarial examples, divergent training, and overfitting. To combat these problems, past research has looked at building neural networks entirely from Lipschitz components. However, these techniques have not matured to the point where researchers have trained a modern architecture such as a transformer with a Lipschitz certificate enforced beyond initialization. To explore this gap, we begin by developing and benchmarking novel, computationally-efficient tools for maintaining norm-constrained weight matrices. Applying these tools, we are able to train transformer models with Lipschitz bounds enforced throughout training. We find that optimizer dynamics matter: switching from AdamW to Muon improves standard methods -- weight decay and spectral normalization -- allowing models to reach equal performance with a lower Lipschitz bound. Inspired by Muon's update having a fixed spectral norm, we co-design a weight constraint method that improves the Lipschitz vs. performance tradeoff on MLPs and 2M parameter transformers. Our 2-Lipschitz transformer on Shakespeare text reaches validation accuracy 60%. Scaling to 145M parameters, our 10-Lipschitz transformer reaches 21% accuracy on internet text. However, to match the NanoGPT baseline validation accuracy of 39.4%, our Lipschitz upper bound increases to 10^264. Nonetheless, our Lipschitz transformers train without stability measures such as layer norm, QK norm, and logit tanh softcapping." +W7221787881,10.48550/arxiv.2610.17969,10.48550/arxiv.2610.17969,2026,Neural Dimensional Analysis: Foundations of Dimensionless and Scale-Consistent Computation,Bum Jun Kim,4,arXiv (Cornell University),0,"Modern Transformers rely on many scale-control mechanisms, including normalization, residual scaling, attention temperatures, loss weighting, and adaptive updates. To explain how these mechanisms maintain scale consistency, we introduce Neural Dimensional Analysis (NDA), a unified framework based on dimensional consistency. We define neural units that specify how tensors rescale and which quantities can be added, just as physical units do in physics. Our rules match residual units, nondimensionalize attention and gate inputs, combine losses across units, and convert gradient units into parameter units. We implement NDA through dimensionless Unit Violation Indices (UVI) for diagnosing scale mismatches within Transformers, a unit-consistent multi-loss objective, and the Gauge-Canonical Transformer (GCT) with normalized coordinates and orthogonalized residual additions. In language-model experiments spanning 51M to 1B parameters, GCT remains stable during training across a broad sixteenfold learning-rate range, outperforming Pre-LN, LLaMA-style, and QK-Norm in 124M validation loss at tuned learning rates. GCT's normalized residual stream also shows improved robustness to quantization. At 1B parameters, calibrated per-tensor int8 quantization increases GCT's validation loss by only 0.009 nats, compared with up to 5.51 nats for the baselines. Vision and physics-informed neural network experiments further confirm NDA's benefits beyond language modeling. These results establish dimensional consistency as a neural network design principle." +W7213292946,10.48550/arxiv.2609.14715,10.48550/arxiv.2609.14715,2026,Depth and Scale in the Sub-150M Regime: JugnuLM-53M vs JugnuLM-110M,Dushyant Rajput,3,arXiv (Cornell University),0,"We scale our conventional sub-150M pretraining recipe from 53.5M to 109.7M parameters, holding the method fixed (Qwen3-style decoder with grouped-query attention, RoPE, SwiGLU, RMSNorm, QK-Norm, and a z-loss; FineWeb-Edu data) and changing only the geometry to a deep-and-thin 23-layer x 576-hidden design. The larger model improves across the board -- BLiMP 78.1 -> 81.3, ARC-Easy 51.4 -> 52.5, WikiText-2 byte-perplexity 2.04 -> 1.95 -- and its 81.3% BLiMP essentially matches GPT-X2-125M (81.28) at about 12% fewer parameters. Notably the 110M model achieves this on fewer training tokens (about 8B vs 12B), so the gain is attributable to capacity and depth, not more data. Both models are deliberately conventional; this report is a clean scaling control and the baseline rung (R0) of an ablation study of what further improves models in this regime. An ablation ladder follows: value residuals (R1) and the Muon optimizer (R2) lift ARC-Easy by a cumulative +3.6 (52.5 -> 56.1) at a near-flat BLiMP and are kept; a diverse data blend (R3) and two logit-distillation settings (R4a/R4b) are not kept -- honest negatives. R3 pins ARC-Easy to FineWeb-Edu's educational filtering rather than raw diversity; distillation from a 1.7B teacher can reach the class-leading ARC-Easy (56.99, matching GPT-X2-125M) but only at a perplexity cost that dialing KD down then erases -- so R2 remains the best kept stack." +W7134257203,,10.5281/zenodo.18924690,2026,Modernizing GPT-2-Scale Models Under a 10B-Token Training Budget,Hang Lu,1,Zenodo (CERN European Organization for Nuclear Research),0,"This technical report investigates whether modern architectural and optimization strategies improve language modeling performance at the GPT-2 (124M) scale under a strict 10B-token single-pass training budget. Starting from a GPT-2-style configuration, the study introduces Rotary Positional Embeddings (RoPE), SwiGLU activation, RMSNorm, Query-Key normalization, and a hybrid Muon + AdamW optimization scheme while keeping the parameter count fixed. All models are trained under the same 10B-token single-pass setting. The report also includes a controlled learning-rate schedule ablation (Warmup–Plateau–Cosine vs. Warmup–Cosine) and analyzes both final metrics and training dynamics. Results suggest that the modernized GPT-2-scale recipe improves validation efficiency under a fixed token budget, while downstream gains remain modest and schedule-dependent. This upload contains the public PDF version of the paper." +W7160755163,,10.5281/zenodo.20114347,2026,VectraYX-Nano: A 42M-Parameter Spanish Cybersecurity LLM with Native MCP Tool Integration,Juan Salas Santillana,1,Zenodo (CERN European Organization for Nuclear Research),0,"We present VectraYX-Nano, a 41.95M-parameter decoder-onlylanguage model trained from scratch in Spanish for cybersecurity,with a Latin-American regional focus and native tool invocationvia the Model Context Protocol (MCP). The model is built aroundfour contributions. (i) Corpus. VectraYX-Sec-ES, a 170M-tokenSpanish corpus assembled by an eight-VM distributed pipeline at∼$25 USD of cloud compute and partitioned into three curriculumphases: conversational (42M tokens, OpenSubtitles-ES [35] andOASST1 [33]), cybersecurity (118M tokens, NVD [39], Wikipedia-ES, in-house NVD-derived Spanish CVE mirror, security blogs),and offensive-security tooling (10M tokens, ExploitDB, HackTricks,OWASP). (ii) Architecture. A 42M-parameter Transformerdecoder combining Grouped-Query Attention [2], QK-Norm [14],RMSNorm [58], SwiGLU [49], RoPE [52], and a 𝑧-loss auxiliary [11],paired with a domain-balanced 16,384-token byte-fallbackBPE [34, 48] trained on a 50/50 conversational/technical mix-ture. (iii) Curriculum with replay. Continual pre-trainingacross the three phases with a replay buffer [30] mitigatescatastrophic forgetting [19, 32] and yields a monotonic lossdescent (9.80 → 3.17 → 3.00 → 2.16). After SFT (final loss 1.74)on a curriculum-aware mixture of OASST-ES, Alpaca-ES, CVEQ&A, and tool-use traces, the v2 bootstrap-ablation referenceattains a conversational gate of 0.775 ± 0.043 on B5 over 𝑁 = 4seeds (Section 8.6), and a controlled Phase-2 replay sweep over{0, 5, 10, 25, 50}% saturates B5 at ≥ 25% replay (Section 6.8).(iv) Two empirical findings, both N=4. A controlled bootstrap-corpus ablation across v2 (OpenSubs), v4 (mC4-ES [57]), andv6 (60/25/15 OpenSubs/mC4/Wiki) under 𝑁 = 4 seeds exposesa loss-versus-register inversion: the lower-perplexity bootstrapsyield measurably worse conversational behavior (𝑣2 > 𝑣4 > 𝑣6on B5 at every paired seed), indicating that at the nano scale thebootstrap-corpus register dominates downstream chat quality.The B4 (tool-selection) floor of 0.000 on the mixed SFT corpusis a corpus-density artifact, not a capacity gate: rebalancing theSFT mixture to tool-use ratio 1:21 yields VectraYX-Nano v7, thereleased headline configuration, which reaches B4 = 0.230 ± 0.052at 42M parameters (𝑁 = 4 seeds) while retaining B1 = 0.332 ± 0.005and B5 = 0.725 ± 0.130 (Section 8.7); a LoRA [29] replication on a260M from-scratch mid-tier reaches 0.445 ± 0.201. The releasedGGUF [22] artifact is 81 MB in F16 (approximately 20 MB in4-bit quantization), runs at sub-second time-to-first-token oncommodity hardware under llama.cpp [21], and is, to the best ofour knowledge, the first published Spanish-native cybersecurityLLM with end-to-end MCP integration. We release the corpusconstruction recipe, training scripts, configurations, GGUF weights,∗ The author is employed at Globant. Institutional affiliation approval is pending.and the B1–B5 benchmark suite (B1: 500, B2: 200, B3: 100, B4: 200,B5: 314 prompts) for reproducibility." +W7112695331,,,2025,"Duality, Weight Decay, and Metrized Deep Learning","Newhouse, Laker",1,DSpace@MIT (Massachusetts Institute of Technology),0,"The Muon optimizer has shown convincing evidence that it is faster and more scalable than AdamW for deep learning training, setting speed records for training NanoGPT and scaling up to models with 16B parameters. The theory that led to Muon is called metrized deep learning, a method that suggests assigning norms to each part of a neural network. Chapter 1 begins with an accessible explanation of metrized deep learning, including one of its recurring tools: odd polynomial iterations that act directly on singular values. Chapter 2 reviews duality, a way to modify the gradient that seeks to decrease the loss the most while disturbing the model the least. Pedagogically, duality links four popular optimizers—SGD, Adam, Shampoo, and Muon—under a common framework, steepest descent under a norm. Practically, experiments suggest that duality-based optimizers train faster than AdamW and transfer learning rate across width. Chapter 3 develops tools to enforce weight norm constraints during training, conferring provable and upfront Lipschitz guarantees for transformers. We find that optimizer dynamics matter: switching from AdamW to Muon improves standard weight regularization methods—weight decay and spectral normalization—allowing models to reach equal performance with a lower Lipschitz bound. Leveraging that Muon’s update has a fixed spectral norm, we co-design a weight constraint method called spectral cap that improves the Lipschitz vs. performance tradeoff for MLPs and 2M parameter transformers. Our 4-Lipschitz transformer on Shakespeare text reaches validation accuracy 60%. Scaling to 145M parameters, our 600-Lipschitz transformer reaches 21% accuracy on internet text. However, to match the NanoGPT baseline validation accuracy of 39.4%, our Lipschitz upper bound increases to 10^274. Nonetheless, our Lipschitz transformers train without stability measures such as layer norm, QK norm, and tanh logit softcapping." +W7162043968,,,2026,A Two-Parameter Weibull Framework for Diagnosing Transformer Weight Distributions,Tiexin Ding,1,arXiv (Cornell University),0,"We apply the Weibull distribution -- a two-parameter family from extreme-value theory -- as a diagnostic framework for element-wise weight magnitude distributions in transformers. At initialization, i.i.d. Gaussian weights give |w| ~ HalfNormal, yielding k ~ 1.20 via middle-80% probability-plot fit (the protocol used throughout this work). This anchor makes k a principled, architecture-independent measuring stick for training dynamics; fitting each weight matrix independently at every layer at every checkpoint enables per-component, per-layer, and per-step diagnostics that aggregate statistics cannot resolve. Applying this framework to 12 model entries spanning 7 architectural families (Pythia, OLMo-1/2, LLaMA-3, Mistral, Qwen2.5/3) reveals three findings. First, FFN modules and the attention output projection W_o -- the Transmission Class -- fall in a narrow k band: median terminal k in [1.186, 1.204] across 12 entries (cross-family CV = 0.51%), shared across SwiGLU/GeLU activations, Pre-LN/QK-Norm placements, and 70M-14B sizes. Second, the attention input projections W_q, W_k -- the Selection Class -- depart from the Weibull family, with severity shaped by storage: separately-stored Q/K (OLMo-1, OLMo-2) yields k in [0.76, 0.99] (deep); GQA models yield k in [1.10, 1.16] (mild); Pythia's merged W_qkv occupies a transitional zone tracking training budget T/tau monotonically. Third, lambda grows substantially during training and scales with sqrt(eta/lambda_wd) within the Pythia family (Pearson r = 0.94, three Transmission kinds), directionally consistent with Fan et al. (2025). The two parameters carry independent information: k labels the functional class, lambda labels training progress. We release npm-weibull-py v0.4 (Python library) and DATABASE_v9_1 at https://github.com/tiexinding/NPM-Weibull-public ." +W7161786876,10.48550/arxiv.2605.18898,10.48550/arxiv.2605.18898,2026,A Two-Parameter Weibull Framework for Diagnosing Transformer Weight Distributions,Tiexin Ding,1,arXiv (Cornell University),0,"We apply the Weibull distribution -- a two-parameter family from extreme-value theory -- as a diagnostic framework for element-wise weight magnitude distributions in transformers. At initialization, i.i.d. Gaussian weights give |w| ~ HalfNormal, yielding k ~ 1.20 via middle-80% probability-plot fit (the protocol used throughout this work). This anchor makes k a principled, architecture-independent measuring stick for training dynamics; fitting each weight matrix independently at every layer at every checkpoint enables per-component, per-layer, and per-step diagnostics that aggregate statistics cannot resolve. Applying this framework to 12 model entries spanning 7 architectural families (Pythia, OLMo-1/2, LLaMA-3, Mistral, Qwen2.5/3) reveals three findings. First, FFN modules and the attention output projection W_o -- the Transmission Class -- fall in a narrow k band: median terminal k in [1.186, 1.204] across 12 entries (cross-family CV = 0.51%), shared across SwiGLU/GeLU activations, Pre-LN/QK-Norm placements, and 70M-14B sizes. Second, the attention input projections W_q, W_k -- the Selection Class -- depart from the Weibull family, with severity shaped by storage: separately-stored Q/K (OLMo-1, OLMo-2) yields k in [0.76, 0.99] (deep); GQA models yield k in [1.10, 1.16] (mild); Pythia's merged W_qkv occupies a transitional zone tracking training budget T/tau monotonically. Third, lambda grows substantially during training and scales with sqrt(eta/lambda_wd) within the Pythia family (Pearson r = 0.94, three Transmission kinds), directionally consistent with Fan et al. (2025). The two parameters carry independent information: k labels the functional class, lambda labels training progress. We release npm-weibull-py v0.4 (Python library) and DATABASE_v9_1 at https://github.com/tiexinding/NPM-Weibull-public ." +W7151240155,,10.1109/ieeeconf68869.2026.11461110,2026,Efficient Russian-English Neural Machine Translation with Architectural Optimizations,German Berezin,2,Systems of Signals Generating and Processing in the Field of on Board Communications,0,"This work addresses efficient neural machine translation deployment for Russian-English translation in resource-constrained onboard communication environments. We propose an enhanced Transformer architecture integrating three key optimizations: Root Mean Square Normalization (RMSNorm) for reduced computational overhead, Query-Key normalization (QKNorm) for robust attention mechanisms, and SwiGLU activation functions for improved parameter efficiency. The approach is evaluated on the OPUS 100 Russian-English parallel corpus using BLEU scores as the primary evaluation metric. The proposed 32-million-parameter model achieves 11.05 BLEU, representing an 8.07 BLEU point improvement over the baseline Transformer with identical parameter count, and demonstrates 3.71x higher parameter efficiency. Ablation studies indicate that architectural design choices significantly impact translation performance within fixed computational budgets. The enhanced architecture enables efficient neural translation deployment on resource-constrained aerospace, maritime, and satellite communication platforms. This work presents a novel architectural combination specifically optimized for Russian-English translation, offering practical insights for deployment of neural translation systems in specialized computing environments." +W7161249368,10.48550/arxiv.2605.13989,10.48550/arxiv.2605.13989,2026,VectraYX-Nano: A 42M-Parameter Spanish Cybersecurity Language Model with Curriculum Learning and Native Tool Use,Juan S. Santillana,1,arXiv (Cornell University),0,"We present VectraYX-Nano, a 41.95M-parameter decoder-only language model trained from scratch in Spanish for cybersecurity, with a Latin-American regional focus and native tool invocation via the Model Context Protocol (MCP). The model has four contributions. (i) Corpus: VectraYX-Sec-ES, a 170M-token Spanish corpus assembled by an eight-VM distributed pipeline at ~$25 USD of cloud compute and split into three curriculum phases (conversational 42M, cybersecurity 118M, offensive tooling 10M). (ii) Architecture: a 42M Transformer decoder with GQA, QK-Norm, RMSNorm, SwiGLU, RoPE and z-loss, paired with a domain-balanced 16,384-token byte-fallback BPE. (iii) Curriculum with replay across the three phases yields a monotonic loss descent (9.80 -> 3.17 -> 3.00 -> 2.16); after SFT (loss 1.74) the v2 bootstrap-ablation reference attains a conversational gate of 0.775 +/- 0.043 on B5 over N=4 seeds, and a controlled Phase-2 replay sweep over {0,5,10,25,50}% saturates B5 at >=25% replay. (iv) Two empirical findings, both N=4. A controlled bootstrap-corpus ablation across v2 (OpenSubs), v4 (mC4-ES), and v6 (60/25/15 OpenSubs/mC4/Wiki) exposes a loss-versus-register inversion: lower-perplexity bootstraps yield measurably worse conversational behavior (v2 > v4 > v6 on B5 at every paired seed). The B4 (tool-selection) floor of 0.000 is a corpus-density artifact, not a capacity gate: rebalancing the SFT mixture to tool-use ratio 1:21 yields VectraYX-Nano v7, the released headline configuration, reaching B4 = 0.230 +/- 0.052 at 42M while retaining B1 = 0.332 +/- 0.005 and B5 = 0.725 +/- 0.130; a LoRA replication on a 260M from-scratch mid-tier reaches 0.445 +/- 0.201. The released GGUF is 96 MB in F16, runs sub-second TTFT on commodity hardware under llama.cpp, and is, to our knowledge, the first published Spanish-native cybersecurity LLM with end-to-end MCP integration." +W7163591057,10.48550/arxiv.2606.04405,10.48550/arxiv.2606.04405,2026,Low-Rank Decay for Grokking in Scale-Invariant Transformers: A Spectral-Geometric View,Mingyu Li,1,arXiv (Cornell University),0,"Modern Transformer architectures frequently employ normalization mechanisms such as RMSNorm and Query-Key Normalization, making parts of the model approximately scale-invariant with respect to weight magnitudes. In this regime, standard Frobenius-norm weight decay acts purely along the radial direction of the weight space and cannot directly simplify the function represented by the normalized layer. We study grokking in small algorithmic tasks through this lens and propose \emph{Low-Rank Decay} (LRD), a nuclear-norm-like spectral regularizer whose subgradient -- the polar factor $UV^\top$ -- retains a tangential component even in the scale-invariant setting. This distinction has a concrete dynamical consequence: after the model memorizes the training set and task gradients vanish, L2 decay can no longer reshape the weight spectrum, whereas LRD continues to compress singular values in an $\ell_1$-like fashion. On modular arithmetic tasks, we find that LRD induces rapid effective-rank collapse in Query/Key matrices and expands the data-fraction boundary at which delayed generalization (grokking) occurs. We further provide a spectral-geometric interpretation through the ``needle-to-fan'' expansion of the nuclear-norm subdifferential near low-rank strata." +W7196970926,,10.52202/085713-2786,2025,"Attention Sinks: A 'Catch, Tag, Release' Mechanism for Embeddings",Stephen Zhang,3,neural information processing systems,0,"Large language models (LLMs) often concentrate their attention on a few specific tokens referred to as attention sinks. Common examples include the first token, a prompt-independent sink, and punctuation tokens, which are prompt-dependent. While the tokens causing the sinks often lack direct semantic meaning, the presence of the sinks is critical for model performance, particularly under model compression and KV-caching. Despite their ubiquity, the function, semantic role, and origin of attention sinks -- especially those beyond the first token -- remain poorly understood. In this work, we conduct a comprehensive investigation demonstrating that attention sinks: catch a sequence of tokens, tag them using a common direction in embedding space, and release them back into the residual stream, where tokens are later retrieved based on the tags they have acquired. Probing experiments reveal these tags carry semantically meaningful information, such as the truth of a statement. These findings extend to reasoning models, where the mechanism spans more heads and explains greater variance in embeddings, or recent models with query-key normalization, where sinks remain just as prevalent. To encourage future theoretical analysis, we introduce a minimal problem which can be solved through the 'catch, tag, release' mechanism, and where it emerges through training." +W4406755454,10.48550/arxiv.2501.12352,10.48550/arxiv.2501.12352,2025,Test-time regression: a unifying framework for designing sequence models with associative memory,Ke Alexander Wang,3,arXiv (Cornell University),0,"Sequence models lie at the heart of modern deep learning. However, rapid advancements have produced a diversity of seemingly unrelated architectures, such as Transformers and recurrent alternatives. In this paper, we introduce a unifying framework to understand and derive these sequence models, inspired by the empirical importance of associative recall, the capability to retrieve contextually relevant tokens. We formalize associative recall as a two-step process, memorization and retrieval, casting memorization as a regression problem. Layers that combine these two steps perform associative recall via ``test-time regression'' over its input tokens. Prominent layers, including linear attention, state-space models, fast-weight programmers, online learners, and softmax attention, arise as special cases defined by three design choices: the regression weights, the regressor function class, and the test-time optimization algorithm. Our approach clarifies how linear attention fails to capture inter-token correlations and offers a mathematical justification for the empirical effectiveness of query-key normalization in softmax attention. Further, it illuminates unexplored regions within the design space, which we use to derive novel higher-order generalizations of softmax attention. Beyond unification, our work bridges sequence modeling with classic regression methods, a field with extensive literature, paving the way for developing more powerful and theoretically principled architectures." +W7222457667,10.48550/arxiv.2610.22330,10.48550/arxiv.2610.22330,2026,Designing Transformers for Continuous Diffusion Language Models,Weijian Luo,5,arXiv (Cornell University),0,"Continuous diffusion language models (CDLMs) offer an alternative approach to parallel language generation; however, most existing studies directly use transformer architectures from either causal large language models (LLMs) or Diffusion Transformers (DiTs) for image generation, which were not optimized for CDLMs. In this paper, we conduct a systematic study on Transformer architectures for CDLMs and introduce Language Diffusion Transformer (LangDiT), combining feature and query-key normalization, wider noise conditioning, likelihood-shortcut gradient routing and calibration, and confidence-gated training self-conditioning. LangDiT significantly reduces generation perplexity in matched LangFlow comparisons, establishing a new state of the art in continuous language generation across different model formulations. Using LangFlow as a representative CDLM, LangDiT as a plug-and-play backbone leads to a new state-of-the-art model with a generation perplexity (GenPPL) of 21.72 using 1024 NFEs and 44.72 GenPPL using 128 NFEs on the unconditional generation of 1024-token sequences on the OpenWebText dataset. When adapted to Plaid-style CDLM formulations, LangDiT results in a new leading GenPPL of 11.33 using 1024 denoising NFEs, and 27.51 GenPPL using 128 denoising NFEs. These gains are achieved while retaining each model's underlying representation, training objective, and inference sampler, isolating transformer architecture as a fundamental source of improvement. Our study identifies architectural design as an important source of improvement in continuous diffusion language models." +W4403364498,10.48550/arxiv.2410.07563,10.48550/arxiv.2410.07563,2024,PLaMo-100B: A Ground-Up Language Model Designed for Japanese Proficiency,"Elements, Preferred",21,arXiv (Cornell University),0,"We introduce PLaMo-100B, a large-scale language model designed for Japanese proficiency. The model was trained from scratch using 2 trillion tokens, with architecture such as QK Normalization and Z-Loss to ensure training stability during the training process. Post-training techniques, including Supervised Fine-Tuning and Direct Preference Optimization, were applied to refine the model's performance. Benchmark evaluations suggest that PLaMo-100B performs well, particularly in Japanese-specific tasks, achieving results that are competitive with frontier models like GPT-4. The base model is available at https://huggingface.co/pfnet/plamo-100b." +W7223737153,,10.5281/zenodo.23179058,2026,"Rounding error in FlashAttention: centered, uncentered, and radial",Hanyu Yang,1,Zenodo (CERN European Organization for Nuclear Research),0,"FlashAttention computes attention one block of keys at a time and divides by the softmax normalizer only at the end. How large is its rounding error, and is it worse than that of ordinary two-pass attention? This note bounds the forward error of FlashAttention-2 for one query, compares it with two-pass attention, and measures the one place where it really goes wrong: the split-KV (Flash-Decoding) merge. Main results One identity does most of the work. An error that hits a key's weight in both the numerator and the normalizer costs only the spread of the values around the output, not their size. Most of FlashAttention's roundings are of this kind. No worst-case winner. FlashAttention-2 and two-pass attention share the same leading term: the cast of the weights to bf16 or fp16. Peaked attention favours FlashAttention-2. The key at the running maximum has weight exactly 1 and casts without error; two-pass casts 1/ℓ, which is rarely representable. The split-KV merge breaks the structure. Its weights are never renormalized, so rounding the global log-sum-exp scales the whole output, and the error grows with the logits. Renormalizing the merge removes it; FlashAttention-3, FlashInfer, vLLM and xformers already merge this way. A published bound fails. The O(u log n) bound of Hsu et al. (ELSA, Theorem 1) is false as stated; Section 6 gives counterexamples from cancelling values and from large logits. Measured on an NVIDIA L4 (PyTorch 2.11 FlashAttention-2, fp16, d = 64, τ = 1/8, identical keys, all values 1, so the exact output is 1). Largest |ŷ − 1| in each logit range, in units of 2−11: batch × keyskernelτs < 27[27, 212)[212, 216)τs ≥ 216 24 × 256single pass0000 24 × 1024split-KV, T = 400≤ 1≤ 24 24 × 2048split-KV, T = 400≤ 1≤ 24 1 × 4096split-KV, T = 1600≤ 2≤ 46 The worst case, 46 × 2−11, is a 2.2% error on an output that should be exactly 1. The figure file plots all 160 logit levels per shape; the CSV holds the raw measurements. Open question. The split-KV error survives the final fp16 cast only once τ max s reaches the thousands. Do models without QK-norm, or models whose logits grow late in training, reach that range during decoding? Also on alphaXiv: the paper and an illustrated companion post. Files. note.pdf (the paper, with full proofs in the appendix); fig_l4_splitkv_error.png (the measurements above, per logit level); l4_splitkv_levels.csv (640 rows: shape, logit level, measured error, and the bound where it applies)." +W7211947725,,10.1007/978-3-032-30438-4_3,2026,Developing GlassGPT: A Modern Day GPT-2 Alternative,Arjun G. Ravi,4,Lecture notes in networks and systems,0, +W7149007078,10.48550/arxiv.2604.01683,10.48550/arxiv.2604.01683,2026,Invertible Query-Key Coupling Composes with Attention Mechanisms,Barak Gahtan,2,arXiv (Cornell University),0,"Scaled dot-product attention forms its queries and keys as independent linear projections, so the two never interact before the dot product that scores them. We study coupled query-key dynamics, a pre-scoring transformation that evolves each token's query and key jointly through a shared invertible coupling before standard scoring. We realize it as an alternating affine map in the style of real non-volume-preserving flows: the coupling is the identity at initialization, adds a small fraction of parameters per head, and leaves the softmax and surrounding architecture unchanged. We place coupling on top of existing attention methods rather than replacing them, and ask whether that composition helps. On WikiText-103, adding coupling to Differential Attention improves on it at both 150M and 455M parameters. At 455M the gain is significant at sequence length 512 (p=0.003, six seeds), survives a Bonferroni correction and replicates on a held-out test split; it also holds across rotary-embedding training lengths 512, 1024, and 2048. The same additive direction appears when coupling is added to query-key normalization (significant at 150M) and Multi-Token Attention (directional). Matched controls attribute the gain to the joint pre-scoring coupling rather than to added capacity, and show that removing the invertibility guarantee preserves the 455M gain yet is far worse than the base method at 150M, so invertibility is what makes the coupling reliable across scales. On its own, coupling lowers perplexity at 60M and 150M (one-sided Welch tests, p<0.05) but the gain narrows with scale and is not significant at 455M. We relate the construction to the expressivity of coupling flows, use an associative-recall study to map where coupling helps and where it degrades sharp retrieval, and conclude that coupling is most useful in composition with a scoring-stage method rather than as a standalone change." +W4414848277,10.48550/arxiv.2502.17092,10.48550/arxiv.2502.17092,2025,Shakti-VLMs: Scalable Vision-Language Models for Enterprise AI,Syed Abdul Gaffar Shakhadri,3,arXiv (Cornell University),0,"We introduce Shakti VLM, a family of vision-language models in the capacity of 1B and 4B parameters designed to address data efficiency challenges in multimodal learning. While recent VLMs achieve strong performance through extensive training data, Shakti models leverage architectural innovations to attain competitive results with fewer tokens. Key advancements include QK-Normalization for attention stability, hybrid normalization techniques, and enhanced positional encoding. A three-stage training strategy further optimizes learning efficiency. Evaluations show that Shakti-Shakti-VLM-1B and Shakti-VLM-4B excel in document understanding, Visual Reasoning, OCR extraction, and general multimodal reasoning. Our results highlight that high performance can be achieved through model design and training strategy rather than sheer data volume, making Shakti an efficient solution for enterprise-scale multimodal tasks." +W7124649764,,10.5281/zenodo.18293025,2026,Complexity-Deep: Token-Routed MLP with Mu-Guided Dynamics for Efficient Transformer Architectures,Boris Peyriguere,1,Zenodo (CERN European Organization for Nuclear Research),0,"We present Complexity-Deep, a transformer architecture introducing two key innovations: (1) Token-Routed MLP, a deterministic expert routing mechanism based on token identity that achieves perfect load balancing without auxiliary losses, and (2) Mu-Guided Dynamics, a simplified PID-inspired system that accumulates context across layers and influences both attention and expert routing. Our Token-Routed MLP uses modulo-based routing (expert_id = token_id mod N) ensuring uniform expert utilization regardless of token frequency distribution. We further introduce CGGR (Contiguous Group GEMM Routing), a Triton-based kernel optimization achieving 5-6x speedup over batched matrix multiplication. The architecture incorporates modern techniques including Grouped Query Attention (GQA), RoPE positional embeddings, QK Normalization, and Flash Attention via SDPA. Our 1.5B parameter model demonstrates the viability of deterministic routing as an alternative to learned routing in Mixture-of-Experts architectures." +W7212826526,10.48550/arxiv.2609.12330,10.48550/arxiv.2609.12330,2026,Unleashing the Power of Equality Saturation for Tensor Program Superoptimization,Qi Zhan,4,arXiv (Cornell University),0,"Efficient GPU implementations of tensor programs often require joint optimization of high-level algebraic formulations and low-level execution strategies. However, the resulting search space grows rapidly as transformations combine across operators, making joint optimization difficult to scale. We present EqiForge, a tensor program superoptimizer based on equality saturation. Its unified IR represents high-level tensor expressions and tiled computations in a single expression language. By composing equality rules, EqiForge derives fused implementations such as FlashAttention-style kernels directly from tensor expressions. Early compaction prunes redundant partial programs before completion, while subgraph composition extends the search to larger graphs. Across tensor-program benchmarks, EqiForge achieves a geometric mean speedup of 1.32x and a maximum of 2.74x over the fastest available baseline per configuration. Its attention kernels outperform FlashAttention by up to 1.87x in decode and approach its performance in prefill. EqiForge also discovers new implementations that outperform torch.compile on various Transformer layers, including QK-normalized MLA (3.16x) and mHC (5.84x)." +W7214945532,10.48550/arxiv.2609.37852,10.48550/arxiv.2609.37852,2026,Delta-Matching: Closing the Final Gap of Native 8-bit Training for LLMs,Haozhan Tang,5,arXiv (Cornell University),0,"Reliable FP8 attention remains a barrier to fully native 8-bit large language model training. We derive how forward-backward inconsistencies produce stale delta and empirically show how it distorts training dynamics. Our stale-delta hybrid runs show a modest loss gap at 569M parameters but substantial loss increases and downstream degradation at 1.67B and 5.29B. QK normalization, NoPE (no positional encoding), and lower-learning-rate context extension mitigate or delay degradation without eliminating it. This pattern suggests accumulated optimization error that smaller models and short runs can conceal. We propose Delta-Matching, proving that it restores the softmax gradient's zero-row-sum invariant under the stated numerical assumptions. It enables native block-scaled FP8 in every forward and backward attention-core matmul without architectural changes, smaller global batches, or auxiliary forward outputs. Across tested architectures, scales, and training stages, Delta-Matching matches BF16/FP32 mixed-precision training loss and overall downstream performance. We will release our implementation, trained models, and data recipes." +W7196936801,,10.52202/085713-2546,2025,Learning in Compact Spaces with Approximately Normalized Transformer,Jörg K. H. Franke,6,neural information processing systems,0,"The successful training of deep neural networks requires addressing challenges such as overfitting, numerical instabilities leading to divergence, and increasing variance in the residual stream. A common solution is to apply regularization and normalization techniques that usually require tuning additional hyperparameters. An alternative is to force all parameters and representations to lie on a hypersphere. This removes the need for regularization and increases convergence speed, but comes with additional costs. In this work, we propose a more holistic, approximate normalization via simple scalar multiplications motivated by the tight concentration of the norms of high-dimensional random vectors. Additionally, instead of applying strict normalization for the parameters, we constrain their norms. These modifications remove the need for weight decay and learning rate warm-up as well, but do not increase the total number of normalization layers. Our experiments with transformer architectures show up to 40% faster convergence compared to GPT models with QK normalization, with only 3% additional runtime cost. When deriving scaling laws, we found that our method enables training with larger batch sizes while preserving the favorable scaling characteristics of classic GPT architectures." +W7223564422,,10.5281/zenodo.23541766,2026,"The Transformer Abides: What Attention Needs, and Why It Endures",Neil D. B. Bruce,1,Zenodo (CERN European Organization for Nuclear Research),0,"Nine years after its introduction, the transformer remains the default sequence model. The usual explanations are real: parallel matrix multiplication on accelerators, predictable returns to scale, a co-evolved ecosystem and exact recall from an uncompressed context. But they do not explain why most modifications barely move accuracy, which properties are essential, why the changes practice adopted took the forms they did, or where the design can still move. This paper is the entry point to a collection of papers that treat a token-mixing layer as an averaging operator rebuilt from its own input and iterated through depth. Such a layer faces six failure modes (collapse to consensus, contamination of content by routing, runaway feedback, the inability to abstain, dilution as context grows, and the inability to represent direction). Each has a protection, proved within a declared class, and a signature measured at small scale when that protection is removed; for gain control, which protects against runaway feedback, that signature is synthetic. We read softmax attention in a pre-normalized residual block as the one known point where all six protections are cheap at once under the prices of parallel hardware and gradient descent; this is an interpretation, not a measured optimum. Two mechanisms, measured at small scale, help explain why its neighbourhood is flat: trained networks compensate local deficits through spare capacity, and, in ten scored tests, making a protection explicit did not beat a block that already supplies it. The anchor, the residual or input path that protects against collapse to consensus, is the one protection whose loss compensation provably cannot repair: under strict mixing, where repeated averaging contracts token differences toward zero, no extra capacity of bounded gain (bounded sensitivity) brings back what is lost without it, and in small-model ablations nothing else in the block made up for it. This is a proved bound; a trained test that adds capacity is pending. Many changes adopted in the mixing layer and its block since 2017, among them pre-normalization, QK-norm, sinks and output gates, read as re-supplies of the same protections; the account is silent on others, such as changes to the MLP. The room that remains for the design to move is where a protection is patched (met only by a fix that works over a limited range, such as the trained length) or paid at a price that another design could avoid. It includes abstention and retrieval at long context, governed by a two-sided log-N law in the context length (with bounded scores, no fixed rule for abstaining keeps both as the context grows), and bounded memory, whose price for exact recall is a per-head capacity. It also includes decisions that sum over all competitors where reading the largest would do, and the writer of new features (in a transformer, the MLP and the output map), for which the collection has no theory yet of what a good writer computes. Along the way the collection establishes results that stand on their own: attention has an arrow that default statistics of a single operator cannot see; depth is a distribution over path lengths; converged adaptive averaging can amplify changes in its input and overturn correct predictions; and softmax obeys length laws that decisions on maxima escape while the competing scores stay below the decision's threshold. We state what is proved, what is measured, what is interpretation, and what is still pending. This is a working draft, not a final version. It is unpolished in places, incomplete in others, and keeps exploratory material a polished paper would leave out, because that material helps decide what comes next and shows the programme's big picture more fully. Please read it with that in mind: the work will be refined further, and later versions will appear under this record. It belongs to a collection released together on Zenodo: the entry paper The Transformer Abides, the monograph A Spectral Theory of Token Communication, and the papers Attention Sinks at Long Context, The Transformer as a Constrained Equilibrium and Beyond Transformers. This version is the text; the records, the code and the formal proofs follow in a later version." +W7214054473,10.48550/arxiv.2609.23314,10.48550/arxiv.2609.23314,2026,ValueDiff: Value-Geometric KV Cache Eviction for Sink-Suppressed LLMs,Junyoung Park,3,arXiv (Cornell University),0,"Modern LLMs with QK-normalization, gated attention, learned attention sinks, or logit softcapping exhibit weaker persistent attention sinks, on which existing KV cache eviction methods primarily rely. We observe that across these models, weaker sinks co-occur with greater value-vector dispersion relative to key-vector dispersion. Motivated by this value-side dispersion, we present ValueDiff, a value-geometric eviction that ranks tokens by the L2 deviation of their value vectors from the cache mean. The same score arises as the minimal-disturbance eviction under a max-entropy assumption about future attention. We evaluate under fixed cache budgets, with eviction at every block boundary during prefill and at every decoding step during generation. On RULER at a tight 2k token budget, ValueDiff retains 88-99% of dense across seven sink-suppressed models (best on 6 out of 7). On LongBench at the 4k budget, ValueDiff averages 92% retention across sink-suppressed models versus 83% for the strongest prior baseline. On MATH-500, ValueDiff is the strongest non-dense method on every sink-suppressed model tested at the 25% cache budget, outperforming prior methods by up to ~20 points on gated-attention models. Across all three benchmarks, value geometry emerges as the more reliable query-invariant eviction signal for sink-suppressed models." +W7165368044,,10.1007/978-3-032-38224-5_42,2026,QG-MIL: A Gated Transformer Aggregator for Domain-Agnostic Multiple Instance Learning in Medical Imaging,Luca Zedda,7,Lecture notes in computer science,0, +W7197353311,,,2025,Cost-Effective Development of Educational Chatbots: Training and Inference Optimization,Aibek Abilgazym,1,Nazarbayev University Repository (Nazarbayev University),0,"Large Language Models (LLMs) have revolutionized natural language processing tasks and show promise for interactive language learning applications. However, training these models and deploying them on resource-constrained devices (like smartphones or Raspberry Pi) is challenging due to their size and computational demands. This thesis presents an accelerated training pipeline for the GPT-2 124M model – the small- est GPT-2 – enabling it to be trained to competitive performance in under 3 minutes (using 0.8 billion tokens) on high-end hardware, a dramatic reduction compared to the original GPT-2 which required about 40 GB of text and multiple days of training. We achieve this speed-up through a combination of architectural optimizations (e.g. Rotary Positional Embeddings, ReLU² activations, QK-normalization, FlexAttention mechanism, and etc.), a custom Muon optimizer for faster convergence, and strate- gic hyperparameter tuning. The model was trained on the cleaned and deduplicated FineWeb dataset. The resulting model was then quantized to 8-bit, reducing memory by a factor of four with minimal loss in language modeling accuracy. We demonstrate the model’s effectiveness in an educational context, focusing on tasks like interac- tive conversation practice and grammar correction, and deployed on edge devices, including an 8GB Raspberry Pi. Benchmarks show that the optimized 124M model maintains language modelling performance (HellaSwag benchmark) while achieving real-time inference speeds within the limited memory ( 1 GB) e of these devices. This work highlights that with the right optimizations, even moderately sized LLMs can be trained and run off-line on personal devices for language learning, bridging the gap between large cloud-based models and accessible on-device AI tutors." +W7223920549,10.48550/arxiv.2610.34582,10.48550/arxiv.2610.34582,2026,HobbyLM: A Sparse Mixture-of-Experts Model Family on a Hobby Budget,Harish Santhanalakshmi Ganesan,3,arXiv (Cornell University),0,"We describe HobbyLM, a family of small sparse Mixture-of-Experts (MoE) models trained from scratch with a few hundred to a few thousand dollars of rented GPU time. The core is a decoder-only transformer with fine-grained experts, sigmoid routing with auxiliary-loss-free load balancing, grouped-query attention with per-head QK-normalization, and aMuon/AdamW optimizer split. Each of these choices was selected with a single-variable ablation suite at 130M parameters. On this architecture we train a 500M-parameter model (169M active) on 40B FineWeb tokens and post-train it into a chat model, a masked-diffusion language model, a function-calling and computer-use agent, and a single ""omni"" checkpoint that reads text, images, video, sound and speech through frozen encoders and small projectors. We run these models on a laptop CPU with a from-scratch Rust inference engine. Two of the authors then adopted the architecture without architectural changes to build HobbyLM-1B (1.04B total, 305M active parameters), trained on a 100B-token curated mixture with a high-quality anneal. HobbyLM-1B reaches a 47.5 average on seven zero-shot benchmarks, and its builders also release an instruction-tuned model and an on-device macOS computer-use adapter. Finally, we describe HobbyLM-Image, a 333M-parameter flow-matching diffusion transformer that generates 1024x1024 images in a 32x-compressed latent space, together with a dependency-light CPU implementation of the full text-to-image pipeline. We report what worked, what did not, and where the small-scale ceilings lie. We also state for each component which author designed or built it." +W7170035928,10.48550/arxiv.2607.18363,10.48550/arxiv.2607.18363,2026,A Controlled Study of Attention-Only Transformers,Henry Ndubuaku,8,arXiv (Cornell University),0,"Feed-forward networks hold two thirds of a transformer's non-embedding parameters, yet the architecture has not received a necessity test that controls parameters, compute, and depth at once. We pretrain attention-only decoder transformers (Simple Attention Networks, SANs) against standard transformers matched separately for parameter count, training FLOPs, and depth (2 to 48 layers), for up to 105B tokens at 6M to 87M parameters. Deleting feed-forward layers in place is costly: the standard transformer leads by 0.47 nats at matched depth and 0.26 nats at matched FLOPs. Reallocating the freed budget into attention depth closes the gap: at matched parameters the difference is 0.006 nats (0.27 percent of loss), reproducible to one part in ten thousand across seed pairs, shrinking across 5B, 30B, and 105B budgets, and holding near 0.02 nats across a 29x size range. Three measurements localize the remaining gap to parametric recall: attention-only models are better on context-grounded answers and worse where knowledge must come from weights. Weight spectra show why: routing matrices (Q/K) crystallize early, content matrices accumulate rank slowly, and removing feed-forward layers relocates this accumulation to the attention output projection. QK-normalization, not feed-forward layers or residual gating, keeps 48-layer attention-only stacks trainable. The deficit concentrates on low-context query prediction and localizes there entirely by the largest budget. A pre-registered test confirms the account: it predicts a 0.02 to 0.05 nat gap on knowledge-dense web text; a matched pair trained on fineweb-edu measures 0.040. Within the tested regime, attention does the rest." diff --git a/02-literature/search/results/S2-openalex-logit-instability.csv b/02-literature/search/results/S2-openalex-logit-instability.csv new file mode 100644 index 0000000..2bde7b1 --- /dev/null +++ b/02-literature/search/results/S2-openalex-logit-instability.csv @@ -0,0 +1,12 @@ +source_id,arxiv,doi,year,title,first_author,n_authors,venue,citations,abstract +W4416789464,10.48550/arxiv.2511.21377,10.48550/arxiv.2511.21377,2025,Controlling changes to attention logits,Ben Anson,2,arXiv (Cornell University),0,"Stability of neural network weights is critical when training transformer models. The query and key weights are particularly problematic, as they tend to grow large without any intervention. Applying normalization to queries and keys, known as `QK norm', fixes stability issues in practice, but is not always applicable. For example, QK norm is not compatible with Multi Latent Attention (MLA) because QK norm requires full materialization of queries and keys during inference, which is not done in MLA. In this paper we suggest that controlling the changes to logits is important for stability. We show that these changes are controllable by assigning parameter-dependent learning rates to the query and key weights. We find that our cheap intervention allows us to increase the base learning rate of the network, outperform other methods in the MLA setting, and achieve performance competitive with QK norm when using Multi-head Attention." +W4415329074,10.48550/arxiv.2505.15548,10.48550/arxiv.2505.15548,2025,Dense Local Dependencies Induce Attention-Logit Explosion and Training Instability During Long-Sequence Transformer Training,Suvadeep Hajra,1,arXiv (Cornell University),0,"Autoregressive transformer language models frequently exhibit training instability when trained on long sequences, particularly under low-precision arithmetic. Although this instability is often accompanied by attention-logit explosion, its underlying cause remains poorly understood. In this work, we present analytical insights and empirical evidence that dense local dependencies are a major contributor to attention-logit explosion. We demonstrate that dense local dependency patterns yield an effectively high-rank attention structure, which the low-rank parameterization of self-attention can only approximate with increasingly large logits as the sequence length grows. This logit inflation ultimately leads to training instability under low-precision arithmetic. We support this explanation through extensive experiments on synthetic and language modeling tasks. Our results consistently show that attention-logit growth increases with sequence length, is mitigated by increasing the attention dimension, and is substantially reduced by explicitly modeling dense local dependencies. Furthermore, we show that this growth is driven by the density of local dependencies rather than by locality alone. More broadly, our findings suggest that explicitly modeling dense local dependencies constitutes an important design principle for developing stable, efficient, and scalable long-context transformer architectures for autoregressive language modeling." +W4324311371,10.48550/arxiv.2303.06296,10.48550/arxiv.2303.06296,2023,Stabilizing Transformer Training by Preventing Attention Entropy Collapse,Shuangfei Zhai,8,arXiv (Cornell University),13,"Training stability is of great importance to Transformers. In this work, we investigate the training dynamics of Transformers by examining the evolution of the attention layers. In particular, we track the attention entropy for each attention head during the course of training, which is a proxy for model sharpness. We identify a common pattern across different architectures and tasks, where low attention entropy is accompanied by high training instability, which can take the form of oscillating loss or divergence. We denote the pathologically low attention entropy, corresponding to highly concentrated attention scores, as $\textit{entropy collapse}$. As a remedy, we propose $σ$Reparam, a simple and efficient solution where we reparametrize all linear layers with spectral normalization and an additional learned scalar. We demonstrate that $σ$Reparam successfully prevents entropy collapse in the attention layers, promoting more stable training. Additionally, we prove a tight lower bound of the attention entropy, which decreases exponentially fast with the spectral norm of the attention logits, providing additional motivation for our approach. We conduct experiments with $σ$Reparam on image classification, image self-supervised learning, machine translation, speech recognition, and language modeling tasks. We show that $σ$Reparam provides stability and robustness with respect to the choice of hyperparameters, going so far as enabling training (a) a Vision Transformer {to competitive performance} without warmup, weight decay, layer normalization or adaptive optimizers; (b) deep architectures in machine translation and (c) speech recognition to competitive performance without warmup and adaptive optimizers. Code is available at \url{https://github.com/apple/ml-sigma-reparam}." +W7106796243,,10.48448/nht8-e622,2025,Variance Sensitivity Induces Attention Entropy Collapse and Instability in Transformers,Association for Computational Linguistics 2025,3,Underline Science Inc.,0,"Attention-based language models typically rely on the softmax function to convert attention logits into probability distributions. However, this mechanism can lead to attention entropy collapse, where attention is focused on a single token, causing training instability. In this work, we identify the high variance sensitivity of softmax as a primary cause of this collapse. We show that entropy-stable attention mechanisms, which either control or are insensitive to the variance of attention logits, can prevent entropy collapse and enable more stable training. We provide empirical evidence of this effect in both large language models (LLMs) and a small Transformer model composed solely of self-attention and support our findings with theoretical analysis. Moreover, we identify that the concentration of attention probabilities increases the probability matrix norm, leading to a gradient exploding." +W7212276107,10.48550/arxiv.2609.11655,10.48550/arxiv.2609.11655,2026,Musec: MomentUm SpEctral Clipping for Stable Muon-type Training,Zhuanghua Liu,3,arXiv (Cornell University),0,"Muon has emerged as a highly effective optimizer for large language model training, often achieving superior convergence and performance compared with the widely adopted Adam and AdamW optimizers. Nevertheless, Muon is prone to training instability due to its spectral flattening, manifested by loss spikes and unbounded growth of model weights. Existing approaches primarily rely on weight or attention-logit clipping, which require architecture-specific modifications and do not directly address instability across all model components. We propose MomentUm SpEctral Clipping (Musec), which replaces Muon's spectral flattening with spectral clipping: rather than setting all singular values of the momentum matrix to approximately one, Musec clips singular values that exceed a threshold while preserving the underlying spectral structure of the momentum. Our strategy provides an optimizer-level, architecture-agnostic mechanism for stabilizing Muon training. Theoretically, we establish convergence guarantees for Musec in nonconvex nonsmooth stochastic optimization. Practically, we develop Soft Musec, an efficient implementation that uses a smooth spectral saturation function approximated by coupled Newton-Schulz iterations. Empirically, Soft Musec consistently improves training stability over existing Muon variants across a wide range of learning rates and model sizes, remaining stable in settings where existing Muon variants diverge while matching their performance under well-tuned configurations." +W7148294612,10.48550/arxiv.2604.00726,10.48550/arxiv.2604.00726,2026,Exploring Silent Data Corruption as a Reliability Challenge in LLM Training,Anton Altenbernd,3,arXiv (Cornell University),0,"As Large Language Models (LLMs) scale in size and complexity, the consequences of failures during training become increasingly severe. A major challenge arises from Silent Data Corruption (SDC): hardware-induced faults that bypass system-level detection mechanisms. SDC may behave like benign numerical noise, but can also cause harmful gradient corruption that leads to loss spikes, divergence, or stalled progress. This work provides a controlled study of how intermittent SDC affects LLM pretraining. Using targeted fault injection at the level of GPU matrix-multiply instructions, we characterize the sensitivity of different bit positions, kernel functions, and execution stages. Our analysis shows that locally originating faults can produce impactful corruption, including NaN propagation, short-lived spikes in loss, gradient norm, and attention logits, as well as persistent parameter divergence. Building on the observed corruption signatures, we propose a lightweight detection method that identifies potentially harmful parameter updates. Experiments on LLaMA models with 60M, 350M, and 1.3B parameters demonstrate that recomputing the most recent training step upon detection can effectively mitigate the impact of these events." +W7149210485,,,2026,Exploring Silent Data Corruption as a Reliability Challenge in LLM Training,Anton Altenbernd,3,arXiv (Cornell University),0,"As Large Language Models (LLMs) scale in size and complexity, the consequences of failures during training become increasingly severe. A major challenge arises from Silent Data Corruption (SDC): hardware-induced faults that bypass system-level detection mechanisms. SDC may behave like benign numerical noise, but can also cause harmful gradient corruption that leads to loss spikes, divergence, or stalled progress. This work provides a controlled study of how intermittent SDC affects LLM pretraining. Using targeted fault injection at the level of GPU matrix-multiply instructions, we characterize the sensitivity of different bit positions, kernel functions, and execution stages. Our analysis shows that locally originating faults can produce impactful corruption, including NaN propagation, short-lived spikes in loss, gradient norm, and attention logits, as well as persistent parameter divergence. Building on the observed corruption signatures, we propose a lightweight detection method that identifies potentially harmful parameter updates. Experiments on LLaMA models with 60M, 350M, and 1.3B parameters demonstrate that recomputing the most recent training step upon detection can effectively mitigate the impact of these events." +W4414448859,10.48550/arxiv.2508.20577,10.48550/arxiv.2508.20577,2025,MERIT: Maximum-normalized Element-wise Ratio for Language Model Large-batch Training,Yang Luo,6,arXiv (Cornell University),0,"Large-batch training has become a cornerstone in accelerating the training of deep neural networks, yet it poses challenges in optimization and generalization. Existing optimizers like AdamW present performance degradation during language models' large-batch training, due to the information bottleneck in attention layers caused by the sharp increase of max attention logit. While the LAMB optimizer partially addresses this issue, some attention layers still face this issue. The reason is that $l_2$-norm-based trust ratios in LAMB are less effective in directly influencing the max value of query/key weights. Furthermore, the weight-wise trust ratio in LAMB is error-prone as it overlooks relationships of weight values within rows or columns. Building on these observations, we propose a novel optimizer, MERIT, which leverages the max-norm to calculate the trust ratio to constrain the max attention logit more effectively. Moreover, we further construct element-wise trust ratios to provide more robust update scaling by focusing on local weight structures. Extensive experiments of large-batch training across various sizes of GPT-2 models demonstrate the superior performance of MERIT. Notably, during the training of GPT-2 Medium, MERIT enables a 6k batch size without any performance degradation compared to the standard batch size (480) with 48B training tokens. This work highlights the importance of considering the max attention logit and finer-granularity trust ratio in large-batch training. It successfully improves the training stability and paves the way for larger batch usage, enabling faster development and iteration of large language models. Code is available at https://github.com/NUS-HPC-AI-Lab/MERIT." +W7127138976,10.48550/arxiv.2601.22966,10.48550/arxiv.2601.22966,2026,A Unified View of Attention and Residual Sinks: Outlier-Driven Rescaling is Essential for Transformer Training,Zihan Qiu,19,arXiv (Cornell University),0,"We investigate the functional role of emergent outliers in large language models, specifically attention sinks (a few tokens that consistently receive large attention logits) and residual sinks (a few fixed dimensions with persistently large activations across most tokens). We hypothesize that these outliers, in conjunction with the corresponding normalizations (\textit{e.g.}, softmax attention and RMSNorm), effectively rescale other non-outlier components. We term this phenomenon \textit{outlier-driven rescaling} and validate this hypothesis across different model architectures and training token counts. This view unifies the origin and mitigation of both sink types. Our main conclusions and observations include: (1) Outliers function jointly with normalization: removing normalization eliminates the corresponding outliers but degrades training stability and performance; directly clipping outliers while retaining normalization leads to degradation, indicating that outlier-driven rescaling contributes to training stability. (2) Outliers serve more as rescale factors rather than contributors, as the final contributions of attention and residual sinks are significantly smaller than those of non-outliers. (3) Outliers can be absorbed into learnable parameters or mitigated via explicit gated rescaling, leading to improved training performance (average gain of 2 points) and enhanced quantization robustness (1.2 points degradation under W4A4 quantization)." +W4414117222,,10.1109/fuzz62266.2025.11152151,2025,Fuzzy Confidence Attention: Harnessing Token-Level Uncertainty for Robust Transformer Models,Taniya Seth,2,IEEE International Conference on Fuzzy Systems (FUZZ-IEEE),0,"Transformer architectures have transformed natural language processing by capturing intricate dependencies through self-attention. Yet, conventional attention mechanisms lack a means to address token-level uncertainty, leaving models vulnerable to noisy, uncertain or unreliable inputs in real-world applications. In this paper, we propose fuzzy confidence attention—a novel mechanism that integrates token-level confidence estimates into the transformer’s attention framework via fuzzy sets. By mapping token confidence scores to fuzzy membership values using a differentiable, piecewise linear function, our method modulates attention logits to dynamically down-weight unreliable tokens. Experimental evaluations on controlled datasets reveal that our approach systematically reallocates attention, as shown by both qualitative heatmap visualizations and quantitative metrics such as Mean Noisy Token Attention and average KL divergence. These results demonstrate improved robustness and interpretability compared to standard attention mechanisms. FCA thus paves the way for more resilient transformer models in noisy and uncertain environments, with promising avenues for future work in large-scale and multimodal settings." +W7161726329,10.48550/arxiv.2605.18106,10.48550/arxiv.2605.18106,2026,"Symmetry-Compatible Principle for Optimizer Design: Embeddings, LM Heads, SwiGLU MLPs, and MoE Routers",Tim Tsz-Kit Lau,2,arXiv (Cornell University),0,"A striking geometric disparity has long persisted in the practice of deep learning. While modern neural network architectures naturally exhibit rich symmetry and equivariance properties, popular optimizers such as Adam and its variants operate inherently coordinate-wise, rendering them unable to respect the equivariance structures of the parameter space. We address this disparity by introducing a symmetry-compatible principle for optimizer design: the gradient update rule should be equivariant under the symmetry group acting on the corresponding weight block. Following this principle, we first provide a unified perspective on bi-orthogonally equivariant updates for general matrix layers, as employed by stochastic spectral descent, Muon, Scion, and polar gradient methods. More importantly, by moving from orthogonal groups to permutation and shared-shift symmetries, we derive symmetry-compatible optimizers for parameter blocks whose symmetries differ from those of general matrix layers: embedding and LM head matrices, SwiGLU MLP projections, and MoE router matrices. These constructions include one-sided spectral, row-norm, hybrid row-norm/spectral, row-aware, column-aware, centered row-norm, and left-spectral updates. They yield an end-to-end layerwise optimizer stack in which each major matrix-valued parameter class is assigned an update whose equivariance matches its symmetry group. We corroborate this principle through pre-training experiments on dense and sparse MoE language models, including Qwen3-0.6B-style, Gemma 3 1B-style, OLMoE-1B-7B-style, and downsized gpt-oss architectures. Across these experiments, symmetry-compatible update rules consistently improve final validation loss, reduce expert load imbalance in sparse MoE models, and in several cases control final vocabulary-logit growth, improve router stability, and overall training stability over the corresponding AdamW updates." diff --git a/02-literature/search/results/S3-openalex-small-scale.csv b/02-literature/search/results/S3-openalex-small-scale.csv new file mode 100644 index 0000000..cdae551 --- /dev/null +++ b/02-literature/search/results/S3-openalex-small-scale.csv @@ -0,0 +1,2 @@ +source_id,arxiv,doi,year,title,first_author,n_authors,venue,citations,abstract +W4404261099,10.48550/arxiv.2410.16682,10.48550/arxiv.2410.16682,2024,Methods of improving LLM training stability,"Rybakov, Oleg",5,arXiv (Cornell University),2,"Training stability of large language models(LLMs) is an important research topic. Reproducing training instabilities can be costly, so we use a small language model with 830M parameters and experiment with higher learning rates to force models to diverge. One of the sources of training instability is the growth of logits in attention layers. We extend the focus of the previous work and look not only at the magnitude of the logits but at all outputs of linear layers in the Transformer block. We observe that with a high learning rate the L2 norm of all linear layer outputs can grow with each training step and the model diverges. Specifically we observe that QKV, Proj and FC2 layers have the largest growth of the output magnitude. This prompts us to explore several options: 1) apply layer normalization not only after QK layers but also after Proj and FC2 layers too; 2) apply layer normalization after the QKV layer (and remove pre normalization). 3) apply QK layer normalization together with softmax capping. We show that with the last two methods we can increase learning rate by 1.5x (without model divergence) in comparison to an approach based on QK layer normalization only. Also we observe significant perplexity improvements for all three methods in comparison to the baseline model." diff --git a/02-literature/search/results/snowball-back-henry2020.csv b/02-literature/search/results/snowball-back-henry2020.csv new file mode 100644 index 0000000..454c1f1 --- /dev/null +++ b/02-literature/search/results/snowball-back-henry2020.csv @@ -0,0 +1,42 @@ +s2_id,arxiv,doi,year,title,first_author,n_authors,venue,citations,abstract +71a72da632d55b4a9e12ca0b9e35bafe0466e318,2005.09561,,2020,Normalized Attention Without Probability Cage,Oliver Richter,2,arXiv.org,29,"Attention architectures are widely used; they recently gained renewed popularity with Transformers yielding a streak of state of the art results. Yet, the geometrical implications of softmax-attention remain largely unexplored. In this work we highlight the limitations of constraining attention weights to the probability simplex and the resulting convex hull of value vectors. We show that Transformers are sequence length dependent biased towards token isolation at initialization and contrast Transformers to simple max- and sum-pooling - two strong baselines rarely reported. We propose to replace the softmax in self-attention with normalization, yielding a hyperparameter and data-bias robust, generally applicable architecture. We support our insights with empirical results from more than 25,000 trained models. All results and implementations are made available." +17293893e0ef799fc038b5da01fc6baf3b08abdd,2005.02433,10.18653/v1/2020.acl-main.198,2020,Stolen Probability: A Structural Weakness of Neural Language Models,David Demeter,3,Annual Meeting of the Association for Computational Linguistics,37,"Neural Network Language Models (NNLMs) generate probability distributions by applying a softmax function to a distance metric formed by taking the dot product of a prediction vector with all word vectors in a high-dimensional embedding space. The dot-product distance metric forms part of the inductive bias of NNLMs. Although NNLMs optimize well with this inductive bias, we show that this results in a sub-optimal ordering of the embedding space that structurally impoverishes some words at the expense of others when assigning probability. We present numerical, theoretical and empirical analyses which show that words on the interior of the convex hull in the embedding space have their probability bounded by the probabilities of the words on the hull." +1780663e87ebf409c828c8c1962cf7f6f54b6fb9,2004.04418,,2020,On Optimal Transformer Depth for Low-Resource Language Translation,Elan Van Biljon,3,arXiv.org,32,"Transformers have shown great promise as an approach to Neural Machine Translation (NMT) for low-resource languages. However, at the same time, transformer models remain difficult to optimize and require careful tuning of hyper-parameters to be useful in this setting. Many NMT toolkits come with a set of default hyper-parameters, which researchers and practitioners often adopt for the sake of convenience and avoiding tuning. These configurations, however, have been optimized for large-scale machine translation data sets with several millions of parallel sentences for European languages like English and French. In this work, we find that the current trend in the field to use very large models is detrimental for low-resource languages, since it makes training more difficult and hurts overall performance, confirming previous observations. We see our work as complementary to the Masakhane project (""Masakhane"" means ""We Build Together"" in isiZulu.) In this spirit, low-resource NMT systems are now being built by the community who needs them the most. However, many in the community still have very limited access to the type of computational resources required for building extremely large models promoted by industrial research. Therefore, by showing that transformer models perform well (and often best) at low-to-moderate depth, we hope to convince fellow researchers to devote less computational resources, as well as time, to exploring overly large models during the development of these systems." +e7246557d6cc04bebdfbba473c130d323e415ccd,2004.14928,10.18653/v1/2020.emnlp-main.615,2020,Language Model Prior for Low-Resource Neural Machine Translation,Christos Baziotis,3,Conference on Empirical Methods in Natural Language Processing,64,"The scarcity of large parallel corpora is an important obstacle for neural machine translation. A common solution is to exploit the knowledge of language models (LM) trained on abundant monolingual data. In this work, we propose a novel approach to incorporate a LM as prior in a neural translation model (TM). Specifically, we add a regularization term, which pushes the output distributions of the TM to be probable under the LM prior, while avoiding wrong predictions when the TM ""disagrees"" with the LM. This objective relates to knowledge distillation, where the LM can be viewed as teaching the TM about the target language. The proposed approach does not compromise decoding speed, because the LM is used only at training time, unlike previous work that requires it during inference. We present an analysis of the effects that different methods have on the distributions of the TM. Results on two low-resource machine translation datasets show clear improvements even with limited monolingual data." +b9d4d87ea78099da5ba3c4351b0889dc203a9ba2,,,2020,PowerNorm: Rethinking Batch Normalization in Transformers,Sheng Shen,5,International Conference on Machine Learning,95, +b45d656ac8cc2e940609580cf291ee76ffcac20a,2002.04745,,2020,On Layer Normalization in the Transformer Architecture,Ruibin Xiong,10,International Conference on Machine Learning,1694,"The Transformer is widely used in natural language processing tasks. To train a Transformer however, one usually needs a carefully designed learning rate warm-up stage, which is shown to be crucial to the final performance but will slow down the optimization and bring more hyperparameter tunings. In this paper, we first study theoretically why the learning rate warm-up stage is essential and show that the location of layer normalization matters. Specifically, we prove with mean field theory that at initialization, for the original-designed Post-LN Transformer, which places the layer normalization between the residual blocks, the expected gradients of the parameters near the output layer are large. Therefore, using a large learning rate on those gradients makes the training unstable. The warm-up stage is practically helpful for avoiding this problem. On the other hand, our theory also shows that if the layer normalization is put inside the residual blocks (recently proposed as Pre-LN Transformer), the gradients are well-behaved at initialization. This motivates us to remove the warm-up stage for the training of Pre-LN Transformers. We show in our experiments that Pre-LN Transformers without the warm-up stage can reach comparable results with baselines while requiring significantly less training time and hyper-parameter tuning on a wide range of applications." +b03cf6324ecf7a295a4aeae5970c88d1a1c3f336,1912.11637,,2019,Explicit Sparse Transformer: Concentrated Attention Through Explicit Selection,Guangxiang Zhao,6,arXiv.org,166,"Self-attention based Transformer has demonstrated the state-of-the-art performances in a number of natural language processing tasks. Self-attention is able to model long-term dependencies, but it may suffer from the extraction of irrelevant information in the context. To tackle the problem, we propose a novel model called \textbf{Explicit Sparse Transformer}. Explicit Sparse Transformer is able to improve the concentration of attention on the global context through an explicit selection of the most relevant segments. Extensive experimental results on a series of natural language processing and computer vision tasks, including neural machine translation, image captioning, and language modeling, all demonstrate the advantages of Explicit Sparse Transformer in model performance. We also show that our proposed sparse attention method achieves comparable or better results than the previous sparse attention method, but significantly reduces training and testing time. For example, the inference speed is twice that of sparsemax in Transformer model. Code will be available at \url{this https URL}" +40922d386116975853a743b1d810c1e0f03e886a,1911.07013,,2019,Understanding and Improving Layer Normalization,Jing-Jing Xu,5,Neural Information Processing Systems,575,"Layer normalization (LayerNorm) is a technique to normalize the distributions of intermediate layers. It enables smoother gradients, faster training, and better generalization accuracy. However, it is still unclear where the effectiveness stems from. In this paper, our main contribution is to take a step further in understanding LayerNorm. Many of previous studies believe that the success of LayerNorm comes from forward normalization. Unlike them, we find that the derivatives of the mean and variance are more important than forward normalization by re-centering and re-scaling backward gradients. Furthermore, we find that the parameters of LayerNorm, including the bias and gain, increase the risk of over-fitting and do not work in most cases. Experiments show that a simple version of LayerNorm (LayerNorm-simple) without the bias and gain outperforms LayerNorm on four datasets. It obtains the state-of-the-art performance on En-Vi machine translation. To address the over-fitting problem, we propose a new normalization method, Adaptive Normalization (AdaNorm), by replacing the bias and gain with a new transformation function. Experiments show that AdaNorm demonstrates better results than LayerNorm on seven out of eight datasets." +d023e4c652dc21b2068a8527203a70d9eaf195d9,1910.13267,10.18653/v1/2020.acl-main.170,2019,BPE-Dropout: Simple and Effective Subword Regularization,Ivan Provilkov,3,Annual Meeting of the Association for Computational Linguistics,364,"Subword segmentation is widely used to address the open vocabulary problem in machine translation. The dominant approach to subword segmentation is Byte Pair Encoding (BPE), which keeps the most frequent words intact while splitting the rare ones into multiple tokens. While multiple segmentations are possible even with the same vocabulary, BPE splits words into unique sequences; this may prevent a model from better learning the compositionality of words and being robust to segmentation errors. So far, the only way to overcome this BPE imperfection, its deterministic nature, was to create another subword segmentation algorithm (Kudo, 2018). In contrast, we show that BPE itself incorporates the ability to produce multiple segmentations of the same word. We introduce BPE-dropout - simple and effective subword regularization method based on and compatible with conventional BPE. It stochastically corrupts the segmentation procedure of BPE, which leads to producing multiple segmentations within the same fixed BPE framework. Using BPE-dropout during training and the standard BPE during inference improves translation quality up to 2.3 BLEU compared to BPE and up to 0.9 BLEU compared to the previous subword regularization." +10eda4521c032adabaa8e70d6569e17370b29dcd,1910.07467,10.5167/UZH-177483,2019,Root Mean Square Layer Normalization,Biao Zhang,2,Neural Information Processing Systems,2159,"Layer normalization (LayerNorm) has been successfully applied to various deep neural networks to help stabilize training and boost model convergence because of its capability in handling re-centering and re-scaling of both inputs and weight matrix. However, the computational overhead introduced by LayerNorm makes these improvements expensive and significantly slows the underlying network, e.g. RNN in particular. In this paper, we hypothesize that re-centering invariance in LayerNorm is dispensable and propose root mean square layer normalization, or RMSNorm. RMSNorm regularizes the summed inputs to a neuron in one layer according to root mean square (RMS), giving the model re-scaling invariance property and implicit learning rate adaptation ability. RMSNorm is computationally simpler and thus more efficient than LayerNorm. We also present partial RMSNorm, or pRMSNorm where the RMS is estimated from p% of the summed inputs without breaking the above properties. Extensive experiments on several tasks using diverse network architectures show that RMSNorm achieves comparable performance against LayerNorm but reduces the running time by 7%~64% on different models. Source code is available at https://github.com/bzhangGo/rmsnorm." +703685e969fed715e13937c11d7ecc5cc7c4dfd0,1910.05895,10.5281/zenodo.3525484,2019,Transformers without Tears: Improving the Normalization of Self-Attention,Toan Q. Nguyen,2,International Workshop on Spoken Language Translation,288,"We evaluate three simple, normalization-centric changes to improve Transformer training. First, we show that pre-norm residual connections (PRENORM) and smaller initializations enable warmup-free, validation-based training with large learning rates. Second, we propose l2 normalization with a single scale parameter (SCALENORM) for faster training and better performance. Finally, we reaffirm the effectiveness of normalizing word embeddings to a fixed length (FIXNORM). On five low-resource translation pairs from TED Talks-based corpora, these changes always converge, giving an average +1.1 BLEU over state-of-the-art bilingual baselines and a new 32.8 BLEU on IWSLT '15 English-Vietnamese. We ob- serve sharper performance curves, more consistent gradient norms, and a linear relationship between activation scaling and decoder depth. Surprisingly, in the high-resource setting (WMT '14 English-German), SCALENORM and FIXNORM remain competitive but PRENORM degrades performance." +95a251513853c6032bdecebd4b74e15795662986,1906.04341,10.18653/v1/W19-4828,2019,What Does BERT Look at? An Analysis of BERT’s Attention,Kevin Clark,4,BlackboxNLP@ACL,2202,"Large pre-trained neural networks such as BERT have had great recent success in NLP, motivating a growing body of research investigating what aspects of language they are able to learn from unlabeled data. Most recent analysis has focused on model outputs (e.g., language model surprisal) or internal vector representations (e.g., probing classifiers). Complementary to these works, we propose methods for analyzing the attention mechanisms of pre-trained models and apply them to BERT. BERT’s attention heads exhibit patterns such as attending to delimiter tokens, specific positional offsets, or broadly attending over the whole sentence, with heads in the same layer often exhibiting similar behaviors. We further show that certain attention heads correspond well to linguistic notions of syntax and coreference. For example, we find heads that attend to the direct objects of verbs, determiners of nouns, objects of prepositions, and coreferent mentions with remarkably high accuracy. Lastly, we propose an attention-based probing classifier and use it to further demonstrate that substantial syntactic information is captured in BERT’s attention." +a39398f68ae7e042f2ef5009e31b4e6a20fd5736,1906.01787,10.18653/v1/P19-1176,2019,Learning Deep Transformer Models for Machine Translation,Qiang Wang,7,Annual Meeting of the Association for Computational Linguistics,786,"Transformer is the state-of-the-art model in recent machine translation evaluations. Two strands of research are promising to improve models of this kind: the first uses wide networks (a.k.a. Transformer-Big) and has been the de facto standard for development of the Transformer system, and the other uses deeper language representation but faces the difficulty arising from learning deep networks. Here, we continue the line of research on the latter. We claim that a truly deep Transformer model can surpass the Transformer-Big counterpart by 1) proper use of layer normalization and 2) a novel way of passing the combination of previous layers to the next. On WMT’16 English-German and NIST OpenMT’12 Chinese-English tasks, our deep system (30/25-layer encoder) outperforms the shallow Transformer-Big/Base baseline (6-layer encoder) by 0.4-2.4 BLEU points. As another bonus, the deep model is 1.6X smaller in size and 3X faster in training than Transformer-Big." +faadd7d081c8d67e8c2567e8a5579e46cd6b2280,1904.01038,10.18653/v1/N19-4009,2019,"fairseq: A Fast, Extensible Toolkit for Sequence Modeling",Myle Ott,8,North American Chapter of the Association for Computational Linguistics,3444,"fairseq is an open-source sequence modeling toolkit that allows researchers and developers to train custom models for translation, summarization, language modeling, and other text generation tasks. The toolkit is based on PyTorch and supports distributed training across multiple GPUs and machines. We also support fast mixed-precision training and inference on modern GPUs. A demo video can be found at https://www.youtube.com/watch?v=OtgDdWtHvto" +88bd75ce3ce22ed85bf9271877aa85da7b7bb312,1903.00089,10.18653/v1/N19-1388,2019,Massively Multilingual Neural Machine Translation,Roee Aharoni,3,North American Chapter of the Association for Computational Linguistics,562,"Multilingual Neural Machine Translation enables training a single model that supports translation from multiple source languages into multiple target languages. We perform extensive experiments in training massively multilingual NMT models, involving up to 103 distinct languages and 204 translation directions simultaneously. We explore different setups for training such models and analyze the trade-offs between translation quality and various modeling decisions. We report results on the publicly available TED talks multilingual corpus where we show that massively multilingual many-to-many models are effective in low resource settings, outperforming the previous state-of-the-art while supporting up to 59 languages in 116 translation directions in a single model. Our experiments on a large-scale dataset with 103 languages, 204 trained directions and up to one million examples per direction also show promising results, surpassing strong bilingual baselines and encouraging future work on massively multilingual NMT." +d96e56ce3971c2da5738f9d17544441b4c78d883,,10.18653/v1/D19-1632,2019,The FLORES Evaluation Datasets for Low-Resource Machine Translation: Nepali–English and Sinhala–English,Francisco (Paco) Guzmán,8,Conference on Empirical Methods in Natural Language Processing,259,"For machine translation, a vast majority of language pairs in the world are considered low-resource because they have little parallel data available. Besides the technical challenges of learning with limited supervision, it is difficult to evaluate methods trained on low-resource language pairs because of the lack of freely and publicly available benchmarks. In this work, we introduce the FLORES evaluation datasets for Nepali–English and Sinhala– English, based on sentences translated from Wikipedia. Compared to English, these are languages with very different morphology and syntax, for which little out-of-domain parallel data is available and for which relatively large amounts of monolingual data are freely available. We describe our process to collect and cross-check the quality of translations, and we report baseline performance using several learning settings: fully supervised, weakly supervised, semi-supervised, and fully unsupervised. Our experiments demonstrate that current state-of-the-art methods perform rather poorly on this benchmark, posing a challenge to the research community working on low-resource MT. Data and code to reproduce our experiments are available at https://github.com/facebookresearch/flores." +c4744a7c2bb298e4a52289a1e085c71cc3d37bc6,1901.02860,10.18653/v1/P19-1285,2019,Transformer-XL: Attentive Language Models beyond a Fixed-Length Context,Zi-Hang Dai,6,Annual Meeting of the Association for Computational Linguistics,4622,"Transformers have a potential of learning longer-term dependency, but are limited by a fixed-length context in the setting of language modeling. We propose a novel neural architecture Transformer-XL that enables learning dependency beyond a fixed length without disrupting temporal coherence. It consists of a segment-level recurrence mechanism and a novel positional encoding scheme. Our method not only enables capturing longer-term dependency, but also resolves the context fragmentation problem. As a result, Transformer-XL learns dependency that is 80% longer than RNNs and 450% longer than vanilla Transformers, achieves better performance on both short and long sequences, and is up to 1,800+ times faster than vanilla Transformers during evaluation. Notably, we improve the state-of-the-art results of bpc/perplexity to 0.99 on enwiki8, 1.08 on text8, 18.3 on WikiText-103, 21.8 on One Billion Word, and 54.5 on Penn Treebank (without finetuning). When trained only on WikiText-103, Transformer-XL manages to generate reasonably coherent, novel text articles with thousands of tokens. Our code, pretrained models, and hyperparameters are available in both Tensorflow and PyTorch." +cf0545d9f171c8e2d88aed13cd7ad60ff2ab8fb5,,10.18653/v1/P18-1167,2018,How Much Attention Do You Need? A Granular Analysis of Neural Machine Translation Architectures,Tobias Domhan,1,Annual Meeting of the Association for Computational Linguistics,79,"With recent advances in network architectures for Neural Machine Translation (NMT) recurrent models have effectively been replaced by either convolutional or self-attentional approaches, such as in the Transformer. While the main innovation of the Transformer architecture is its use of self-attentional layers, there are several other aspects, such as attention with multiple heads and the use of many attention layers, that distinguish the model from previous baselines. In this work we take a fine-grained look at the different architectures for NMT. We introduce an Architecture Definition Language (ADL) allowing for a flexible combination of common building blocks. Making use of this language we show in experiments that one can bring recurrent and convolutional models very close to the Transformer performance by borrowing concepts from the Transformer architecture, but not using self-attention. Additionally, we find that self-attention is much more important on the encoder side than on the decoder side, where it can be replaced by a RNN or CNN without a loss in performance in most settings. Surprisingly, even a model without any target side self-attention performs well." +6e45251b16cd423f3c025f004959c6d2b26efab0,1805.00631,10.18653/v1/P18-1166,2018,Accelerating Neural Transformer via an Average Attention Network,Biao Zhang,3,Annual Meeting of the Association for Computational Linguistics,129,"With parallelizable attention networks, the neural Transformer is very fast to train. However, due to the auto-regressive architecture and self-attention in the decoder, the decoding procedure becomes slow. To alleviate this issue, we propose an average attention network as an alternative to the self-attention network in the decoder of the neural Transformer. The average attention network consists of two layers, with an average layer that models dependencies on previous positions and a gating layer that is stacked over the average layer to enhance the expressiveness of the proposed attention network. We apply this network on the decoder part of the neural Transformer to replace the original target-side self-attention model. With masking tricks and dynamic programming, our model enables the neural Transformer to decode sentences over four times faster than its original version with almost no loss in training time and translation performance. We conduct a series of experiments on WMT17 translation tasks, where on 6 different language pairs, we obtain robust and consistent speed-ups in decoding." +bb669de2fce407df2f5cb2f8c51dedee3f467e04,1804.09849,10.18653/v1/P18-1008,2018,The Best of Both Worlds: Combining Recent Advances in Neural Machine Translation,Mia Chen,16,Annual Meeting of the Association for Computational Linguistics,482,"The past year has witnessed rapid advances in sequence-to-sequence (seq2seq) modeling for Machine Translation (MT). The classic RNN-based approaches to MT were first out-performed by the convolutional seq2seq model, which was then out-performed by the more recent Transformer model. Each of these new approaches consists of a fundamental architecture accompanied by a set of modeling and training techniques that are in principle applicable to other seq2seq architectures. In this paper, we tease apart the new architectures and their accompanying techniques in two ways. First, we identify several key modeling and training techniques, and apply them to the RNN architecture, yielding a new RNMT+ model that outperforms all of the three fundamental architectures on the benchmark WMT’14 English to French and English to German tasks. Second, we analyze the properties of each fundamental seq2seq architecture and devise new hybrid architectures intended to combine their strengths. Our hybrid models obtain further improvements, outperforming the RNMT+ model on both benchmark datasets." +b4bfadfca9742bb3ee98a0cd322d5ce4e59a3ceb,1804.08771,10.18653/v1/W18-6319,2018,A Call for Clarity in Reporting BLEU Scores,Matt Post,1,Conference on Machine Translation,3729,"The field of machine translation faces an under-recognized problem because of inconsistency in the reporting of scores from its dominant metric. Although people refer to “the” BLEU score, BLEU is in fact a parameterized metric whose values can vary wildly with changes to these parameters. These parameters are often not reported or are hard to find, and consequently, BLEU scores between papers cannot be directly compared. I quantify this variation, finding differences as high as 1.8 between commonly used configurations. The main culprit is different tokenization and normalization schemes applied to the reference. Pointing to the success of the parsing community, I suggest machine translation researchers settle upon the BLEU scheme used by the annual Conference on Machine Translation (WMT), which does not allow for user-supplied reference processing, and provide a new tool, SACREBLEU, to facilitate this." +7b29f45df975ed1e4c3864b6ab4483f11086aa76,1804.06323,10.18653/v1/N18-2084,2018,When and Why Are Pre-Trained Word Embeddings Useful for Neural Machine Translation?,Ye Qi,5,North American Chapter of the Association for Computational Linguistics,379,"The performance of Neural Machine Translation (NMT) systems often suffers in low-resource scenarios where sufficiently large-scale parallel corpora cannot be obtained. Pre-trained word embeddings have proven to be invaluable for improving performance in natural language analysis tasks, which often suffer from paucity of data. However, their utility for NMT has not been extensively explored. In this work, we perform five sets of experiments that analyze when we can expect pre-trained word embeddings to help in NMT tasks. We show that such embeddings can be surprisingly effective in some cases – providing gains of up to 20 BLEU points in the most favorable setting." +642c1b4a9da95ea4239708afc5929a5007a1870d,1803.07416,,2018,Tensor2Tensor for Neural Machine Translation,Ashish Vaswani,13,Conference of the Association for Machine Translation in the Americas,556,Tensor2Tensor is a library for deep learning models that is well-suited for neural machine translation and includes the reference implementation of the state-of-the-art Transformer model. +3861ae2a6bdd2a759c2d901a6583e63a216bc2fc,1711.02132,,2017,Weighted Transformer Network for Machine Translation,Karim Ahmed,3,arXiv.org,143,"State-of-the-art results on neural machine translation often use attentional sequence-to-sequence models with some form of convolution or recursion. Vaswani et al. (2017) propose a new architecture that avoids recurrence and convolution completely. Instead, it uses only self-attention and feed-forward layers. While the proposed architecture achieves state-of-the-art results on several machine translation tasks, it requires a large number of parameters and training iterations to converge. We propose Weighted Transformer, a Transformer with modified attention layers, that not only outperforms the baseline network in BLEU score but also converges 15-40% faster. Specifically, we replace the multi-head attention by multiple self-attention branches that the model learns to combine during the training process. Our model improves the state-of-the-art performance by 0.5 BLEU points on the WMT 2014 English-to-German translation task and by 0.4 on the English-to-French translation task." +3acf8313546def523da334cb3324806ac229769f,1710.01329,10.18653/v1/N18-1031,2017,Improving Lexical Choice in Neural Machine Translation,Toan Q. Nguyen,2,North American Chapter of the Association for Computational Linguistics,90,"We explore two solutions to the problem of mistranslating rare words in neural machine translation. First, we argue that the standard output layer, which computes the inner product of a vector representing the context with all possible output word embeddings, rewards frequent words disproportionately, and we propose to fix the norms of both vectors to a constant value. Second, we integrate a simple lexical module which is jointly trained with the rest of the model. We evaluate our approaches on eight language pairs with data sizes ranging from 100k to 8M words, and achieve improvements of up to +4.3 BLEU, surpassing phrase-based translation in nearly all settings." +204e3073870fae3d05bcbc2f6a8e263d9b72e776,1706.03762,,2017,Attention is All you Need,Ashish Vaswani,8,Neural Information Processing Systems,199000,"The dominant sequence transduction models are based on complex recurrent or convolutional neural networks in an encoder-decoder configuration. The best performing models also connect the encoder and decoder through an attention mechanism. We propose a new simple network architecture, the Transformer, based solely on attention mechanisms, dispensing with recurrence and convolutions entirely. Experiments on two machine translation tasks show these models to be superior in quality while being more parallelizable and requiring significantly less time to train. Our model achieves 28.4 BLEU on the WMT 2014 English-to-German translation task, improving over the existing best results, including ensembles by over 2 BLEU. On the WMT 2014 English-to-French translation task, our model establishes a new single-model state-of-the-art BLEU score of 41.8 after training for 3.5 days on eight GPUs, a small fraction of the training costs of the best models from the literature. We show that the Transformer generalizes well to other tasks by applying it successfully to English constituency parsing both with large and limited training data." +e5aabd29682c8b8526833cbe9182e3e35a0c9ae8,1702.05870,10.1007/978-3-030-01418-6_38,2017,Cosine Normalization: Using Cosine Similarity Instead of Dot Product in Neural Networks,Chun-Jie Luo,6,International Conference on Artificial Neural Networks,253,"Traditionally, multi-layer neural networks use dot product between the output vector of previous layer and the incoming weight vector as the input to activation function. The result of dot product is unbounded, thus increases the risk of large variance. Large variance of neuron makes the model sensitive to the change of input distribution, thus results in poor generalization, and aggravates the internal covariate shift which slows down the training. To bound dot product and decrease the variance, we propose to use cosine similarity or centered cosine similarity (Pearson Correlation Coefficient) instead of dot product in neural networks, which we call cosine normalization. We compare cosine normalization with batch, weight and layer normalization in fully-connected neural networks, convolutional networks on the data sets of MNIST, 20NEWS GROUP, CIFAR-10/100, SVHN. Experiments show that cosine normalization achieves better performance than other normalization techniques." +aab5002a22b9b4244a8329b140bd0a86021aa2d1,1709.03815,10.18653/V1/P17-4012,2017,OpenNMT: Open-Source Toolkit for Neural Machine Translation,G. Klein,6,Annual Meeting of the Association for Computational Linguistics,1986,"We describe an open-source toolkit for neural machine translation (NMT). The toolkit prioritizes efficiency, modularity, and extensibility with the goal of supporting NMT research into model architectures, feature representations, and source modalities, while maintaining competitive performance and reasonable training requirements. The toolkit consists of modeling and translation support, as well as detailed pedagogical documentation about the underlying techniques." +97fb4e3d45bb098e27e0071448b6152217bd35a5,1607.06450,,2016,Layer Normalization,Jimmy Ba,3,arXiv.org,13427,"Training state-of-the-art, deep neural networks is computationally expensive. One way to reduce the training time is to normalize the activities of the neurons. A recently introduced technique called batch normalization uses the distribution of the summed input to a neuron over a mini-batch of training cases to compute a mean and variance which are then used to normalize the summed input to that neuron on each training case. This significantly reduces the training time in feed-forward neural networks. However, the effect of batch normalization is dependent on the mini-batch size and it is not obvious how to apply it to recurrent neural networks. In this paper, we transpose batch normalization into layer normalization by computing the mean and variance used for normalization from all of the summed inputs to the neurons in a layer on a single training case. Like batch normalization, we also give each neuron its own adaptive bias and gain which are applied after the normalization but before the non-linearity. Unlike batch normalization, layer normalization performs exactly the same computation at training and test times. It is also straightforward to apply to recurrent neural networks by computing the normalization statistics separately at each time step. Layer normalization is very effective at stabilizing the hidden state dynamics in recurrent networks. Empirically, we show that layer normalization can substantially reduce the training time compared with previously published techniques." +1518039b5001f1836565215eb047526b3ac7f462,1508.07909,10.18653/v1/P16-1162,2015,Neural Machine Translation of Rare Words with Subword Units,Rico Sennrich,3,Annual Meeting of the Association for Computational Linguistics,9167,"Neural machine translation (NMT) models typically operate with a fixed vocabulary, but translation is an open-vocabulary problem. Previous work addresses the translation of out-of-vocabulary words by backing off to a dictionary. In this paper, we introduce a simpler and more effective approach, making the NMT model capable of open-vocabulary translation by encoding rare and unknown words as sequences of subword units. This is based on the intuition that various word classes are translatable via smaller units than words, for instance names (via character copying or transliteration), compounds (via compositional translation), and cognates and loanwords (via phonological and morphological transformations). We discuss the suitability of different word segmentation techniques, including simple character n-gram models and a segmentation based on the byte pair encoding compression algorithm, and empirically show that subword models improve over a back-off dictionary baseline for the WMT 15 translation tasks English-German and English-Russian by 1.1 and 1.3 BLEU, respectively." +995c5f5e62614fcb4d2796ad2faab969da51713e,1502.03167,,2015,Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift,S. Ioffe,2,International Conference on Machine Learning,48194,"Training Deep Neural Networks is complicated by the fact that the distribution of each layer's inputs changes during training, as the parameters of the previous layers change. This slows down the training by requiring lower learning rates and careful parameter initialization, and makes it notoriously hard to train models with saturating nonlinearities. We refer to this phenomenon as internal covariate shift, and address the problem by normalizing layer inputs. Our method draws its strength from making normalization a part of the model architecture and performing the normalization for each training mini-batch. Batch Normalization allows us to use much higher learning rates and be less careful about initialization, and in some cases eliminates the need for Dropout. Applied to a state-of-the-art image classification model, Batch Normalization achieves the same accuracy with 14 times fewer training steps, and beats the original model by a significant margin. Using an ensemble of batch-normalized networks, we improve upon the best published result on ImageNet classification: reaching 4.82% top-5 test error, exceeding the accuracy of human raters." +cb826a3899752b796f14df1c50378c64954a6b0a,,,2004,Statistical Significance Tests for Machine Translation Evaluation,Philipp Koehn,1,Conference on Empirical Methods in Natural Language Processing,2002, +d7da009f457917aa381619facfa5ffae9329a6e9,,10.3115/1073083.1073135,2002,Bleu: a Method for Automatic Evaluation of Machine Translation,K. Papineni,4,Annual Meeting of the Association for Computational Linguistics,35442,"Human evaluations of machine translation are extensive but expensive. Human evaluations can take months to finish and involve human labor that can not be reused. We propose a method of automatic machine translation evaluation that is quick, inexpensive, and language-independent, that correlates highly with human evaluation, and that has little marginal cost per run. We present this method as an automated understudy to skilled human judges which substitutes for them when there is need for quick or frequent evaluations." +77a02706edbdca6ad63ddf903ebd559dd7d423ac,cs/0205028,10.3115/1118108.1118117,2002,NLTK: The Natural Language Toolkit,E. Loper,2,Annual Meeting of the Association for Computational Linguistics,1472,"The Natural Language Toolkit is a suite of program modules, data sets and tutorials supporting research and teaching in computational linguistics and natural language processing. NLTK is written in Python and distributed under the GPL open source license. Over the past year the toolkit has been rewritten, simplifying many linguistic data structures and taking advantage of recent enhancements in the Python language. This paper reports on the simplified toolkit and explains how it is used in teaching NLP." +7344b71d205a7ad71ebad2873c063d7008f231fc,,,2016,The IWSLT 2016 Evaluation Campaign,M. Cettolo,6,International Workshop on Spoken Language Translation,99, +,,,2016,Hinton,,0,Layer normalization,0, +b54268e3b8d148c0695ca52bebb0f80e26a4b987,,,2015,The IWSLT 2015 Evaluation Campaign,M. Cettolo,6,International Workshop on Spoken Language Translation,165, +6ed9417eaa7ee16f0563599829a061421a3e0563,,10.3115/1225403.1225421,2006,NLTK: The Natural Language Toolkit,Steven Bird,1,Annual Meeting of the Association for Computational Linguistics,5340,"The Natural Language Toolkit is a suite of program modules, data sets and tutorials supporting research and teaching in computational linguistics and natural language processing. NLTK is written in Python and distributed under the GPL open source license. Over the past year the toolkit has been rewritten, simplifying many linguistic data structures and taking advantage of recent enhancements in the Python language. This paper reports on the simpli-fied toolkit and explains how it is used in teaching NLP." +,,,,Like d in scaled dot product,,0,heuristics,0, +,,,,Cosine Similarity Instead of Dot Product,,0,Neural Networks,0, +,,,,"Appendix A Varying the Number of Heads In Table 4, we show the performance of QKN",,0,,0, diff --git a/02-literature/search/results/snowball-back-imu1.csv b/02-literature/search/results/snowball-back-imu1.csv new file mode 100644 index 0000000..65ba59d --- /dev/null +++ b/02-literature/search/results/snowball-back-imu1.csv @@ -0,0 +1,23 @@ +s2_id,arxiv,doi,year,title,first_author,n_authors,venue,citations,abstract +6840c3ec7e97b2c7c65b2b989634ac83f9827d91,2511.18903,10.48550/arXiv.2511.18903,2025,How Learning Rate Decay Wastes Your Best Data in Curriculum-Based LLM Pretraining,Kai-Rong Luo,8,arXiv.org,13,"Due to the scarcity of high-quality data, large language models (LLMs) are often trained on mixtures of data with varying quality levels, even after sophisticated data curation. A natural approach to better leverage high-quality data is curriculum-based pretraining, where the model is trained on data sorted in ascending order of quality as determined by a quality metric. However, prior studies have reported limited improvements from such curriculum-based pretraining strategies. This work identifies a critical factor constraining these methods: the incompatibility between the ascending data quality order and the decaying learning rate (LR) schedule. We find that while curriculum-based training substantially outperforms random shuffling when using a constant LR, its advantage diminishes under standard LR decay schedules. Our experiments show this incompatibility can be mitigated by two simple strategies: (1) employing a more moderate LR decay schedule, where the final LR is only moderately smaller than the peak LR, and (2) replacing LR decay with model averaging, i.e., computing a weighted average of the final few checkpoints. By combining these strategies, we improve the average score on a suite of standard benchmarks by 1.64% over random shuffling, without additional data refinement. Validated on 1.5B-parameter models trained over 30B tokens with various data-quality metrics, our findings call for a re-evaluation of curriculum-based LLM pretraining and underscore the potential of co-designing data curricula with optimization methods." +79378d081d8e975220fc00627a1b86436bbed27c,2510.12402,10.48550/arXiv.2510.12402,2025,Cautious Weight Decay,Li-Zhang Chen,9,arXiv.org,16,"We introduce Cautious Weight Decay (CWD), a one-line, optimizer-agnostic modification that applies weight decay only to parameter coordinates whose signs align with the optimizer update. Unlike standard decoupled decay, which implicitly optimizes a regularized or constrained objective, CWD preserves the original loss and admits a bilevel interpretation: it induces sliding-mode behavior upon reaching the stationary manifold, allowing it to search for locally Pareto-optimal stationary points of the unmodified objective. In practice, CWD is a drop-in change for optimizers such as AdamW, Lion, and Muon, requiring no new hyperparameters or additional tuning. For language model pre-training and ImageNet classification, CWD consistently improves final loss and accuracy at million- to billion-parameter scales." +036d28ef268c1bdee30d2f1a5b69bf1cd54f0323,2510.05491,10.48550/arXiv.2510.05491,2025,NorMuon: Making Muon more efficient and scalable,Zi-Chong Li,5,arXiv.org,88,"The choice of optimizer significantly impacts the training efficiency and computational costs of large language models (LLMs). Recently, the Muon optimizer has demonstrated promising results by orthogonalizing parameter updates, improving optimization geometry through better conditioning. Despite Muon's emergence as a candidate successor to Adam, the potential for jointly leveraging their strengths has not been systematically explored. In this work, we bridge this gap by proposing NorMuon (Neuron-wise Normalized Muon), an optimizer that synergistically combines orthogonalization with neuron-level adaptive learning rates. Our analysis reveals that while Muon effectively reduces condition numbers, the resulting updates exhibit highly non-uniform neuron norms, causing certain neurons to dominate the optimization process. NorMuon addresses this imbalance by maintaining second-order momentum statistics for each neuron and applying row-wise normalization after orthogonalization, ensuring balanced parameter utilization while preserving Muon's conditioning benefits. To enable practical deployment at scale, we develop an efficient distributed implementation under the FSDP2 framework that strategically distributes orthogonalization computations across devices. Experiments across multiple model scales demonstrate that NorMuon consistently outperforms both Adam and Muon, achieving 21.74% better training efficiency than Adam and 11.31% improvement over Muon on 1.1 B pretraining setting, while maintaining a comparable memory footprint to Muon. Our findings suggest that orthogonalization and adaptive learning rates are complementary rather than competing approaches, opening new avenues for optimizer design in large-scale deep learning." +3cf4e1e35bf302b84065cd2a8aa744fd275d5c22,2502.05795,10.48550/arXiv.2502.05795,2025,The Curse of Depth in Large Language Models,Wenfang Sun,6,Neural Information Processing Systems,70,"In this paper, we introduce the Curse of Depth, a concept that highlights, explains, and addresses the recent observation in modern Large Language Models (LLMs) where nearly half of the layers are less effective than expected. We first confirm the wide existence of this phenomenon across the most popular families of LLMs such as Llama, Mistral, DeepSeek, and Qwen. Our analysis, theoretically and empirically, identifies that the underlying reason for the ineffectiveness of deep layers in LLMs is the widespread usage of Pre-Layer Normalization (Pre-LN). While Pre-LN stabilizes the training of Transformer LLMs, its output variance exponentially grows with the model depth, which undesirably causes the derivative of the deep Transformer blocks to be an identity matrix, and therefore barely contributes to the training. To resolve this training pitfall, we propose LayerNorm Scaling (LNS), which scales the variance of output of the layer normalization inversely by the square root of its depth. This simple modification mitigates the output variance explosion of deeper Transformer layers, improving their contribution. Across a wide range of model sizes (130M to 7B), our experiments show that LNS consistently outperforms previous normalization and scaling techniques in enhancing LLM pre-training performance. Moreover, this improvement seamlessly carries over to supervised fine-tuning. All these gains can be attributed to the fact that LayerNorm Scaling enables deeper layers to contribute more effectively during training. Our code is available at \href{https://github.com/lmsdss/LayerNorm-Scaling}{LayerNorm-Scaling}." +b096dc25eeacb00ca35293131230e6234bac9a59,2410.05192,10.48550/arXiv.2410.05192,2024,Understanding Warmup-Stable-Decay Learning Rates: A River Valley Loss Landscape Perspective,Kai-Yue Wen,6,arXiv.org,88,"Training language models currently requires pre-determining a fixed compute budget because the typical cosine learning rate schedule depends on the total number of steps. In contrast, the Warmup-Stable-Decay (WSD) schedule uses a constant learning rate to produce a main branch of iterates that can in principle continue indefinitely without a pre-specified compute budget. Then, given any compute budget, one can branch out from the main branch at a proper time with a rapidly decaying learning rate to produce a strong model. Empirically, WSD generates a non-traditional loss curve: the loss remains elevated during the stable phase but sharply declines during the decay phase. Towards explaining this phenomenon, we conjecture that pretraining loss exhibits a river valley landscape, which resembles a deep valley with a river at its bottom. Under this assumption, we show that during the stable phase, the iterate undergoes large oscillations due to the high learning rate, yet it progresses swiftly along the river. During the decay phase, the rapidly dropping learning rate minimizes the iterate's oscillations, moving it closer to the river and revealing true optimization progress. Therefore, the sustained high learning rate phase and fast decaying phase are responsible for progress in the river and the mountain directions respectively, and are both critical. Our analysis predicts phenomenons consistent with empirical observations and shows that this landscape can emerge from pretraining on a simple bi-gram dataset. Inspired by the theory, we introduce WSD-S, a variant of WSD that reuses previous checkpoints' decay phases and keeps only one main branch, where we resume from a decayed checkpoint. WSD-S empirically outperforms WSD and Cyclic-Cosine in obtaining multiple language model checkpoints across various compute budgets in a single run for parameters scaling from 0.1B to 1.2B." +874e957f6bcbfeb9f69d4475456abb13335ec05b,2406.11794,10.48550/arXiv.2406.11794,2024,DataComp-LM: In search of the next generation of training sets for language models,Jeffrey Li,59,Neural Information Processing Systems,466,"We introduce DataComp for Language Models (DCLM), a testbed for controlled dataset experiments with the goal of improving language models. As part of DCLM, we provide a standardized corpus of 240T tokens extracted from Common Crawl, effective pretraining recipes based on the OpenLM framework, and a broad suite of 53 downstream evaluations. Participants in the DCLM benchmark can experiment with data curation strategies such as deduplication, filtering, and data mixing at model scales ranging from 412M to 7B parameters. As a baseline for DCLM, we conduct extensive experiments and find that model-based filtering is key to assembling a high-quality training set. The resulting dataset, DCLM-Baseline enables training a 7B parameter language model from scratch to 64% 5-shot accuracy on MMLU with 2.6T training tokens. Compared to MAP-Neo, the previous state-of-the-art in open-data language models, DCLM-Baseline represents a 6.6 percentage point improvement on MMLU while being trained with 40% less compute. Our baseline model is also comparable to Mistral-7B-v0.3 and Llama 3 8B on MMLU (63%&66%), and performs similarly on an average of 53 natural language understanding tasks while being trained with 6.6x less compute than Llama 3 8B. Our results highlight the importance of dataset design for training language models and offer a starting point for further research on data curation." +fdc53c2c10742464087c0525f77e32604827a21d,2309.17453,10.48550/arXiv.2309.17453,2023,Efficient Streaming Language Models with Attention Sinks,Guang-Xuan Xiao,5,International Conference on Learning Representations,2726,"Deploying Large Language Models (LLMs) in streaming applications such as multi-round dialogue, where long interactions are expected, is urgently needed but poses two major challenges. Firstly, during the decoding stage, caching previous tokens' Key and Value states (KV) consumes extensive memory. Secondly, popular LLMs cannot generalize to longer texts than the training sequence length. Window attention, where only the most recent KVs are cached, is a natural approach -- but we show that it fails when the text length surpasses the cache size. We observe an interesting phenomenon, namely attention sink, that keeping the KV of initial tokens will largely recover the performance of window attention. In this paper, we first demonstrate that the emergence of attention sink is due to the strong attention scores towards initial tokens as a""sink""even if they are not semantically important. Based on the above analysis, we introduce StreamingLLM, an efficient framework that enables LLMs trained with a finite length attention window to generalize to infinite sequence lengths without any fine-tuning. We show that StreamingLLM can enable Llama-2, MPT, Falcon, and Pythia to perform stable and efficient language modeling with up to 4 million tokens and more. In addition, we discover that adding a placeholder token as a dedicated attention sink during pre-training can further improve streaming deployment. In streaming settings, StreamingLLM outperforms the sliding window recomputation baseline by up to 22.2x speedup. Code and datasets are provided at https://github.com/mit-han-lab/streaming-llm." +5ae6fb6b5a3c7df515ff4a82ac9673bae6a8e200,2305.13245,10.48550/arXiv.2305.13245,2023,GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints,Joshua Ainslie,6,Conference on Empirical Methods in Natural Language Processing,1954,"Multi-query attention (MQA), which only uses a single key-value head, drastically speeds up decoder inference. However, MQA can lead to quality degradation, and moreover it may not be desirable to train a separate model just for faster inference. We (1) propose a recipe for uptraining existing multi-head language model checkpoints into models with MQA using 5% of original pre-training compute, and (2) introduce grouped-query attention (GQA), a generalization of multi-query attention which uses an intermediate (more than one, less than number of query heads) number of key-value heads. We show that uptrained GQA achieves quality close to multi-head attention with comparable speed to MQA." +61e721334296ebfbbf6443b5ed9eb8c83b708c95,2302.05442,10.48550/arXiv.2302.05442,2023,Scaling Vision Transformers to 22 Billion Parameters,Mostafa Dehghani,42,International Conference on Machine Learning,972,"The scaling of Transformers has driven breakthrough capabilities for language models. At present, the largest large language models (LLMs) contain upwards of 100B parameters. Vision Transformers (ViT) have introduced the same architecture to image and video modelling, but these have not yet been successfully scaled to nearly the same degree; the largest dense ViT contains 4B parameters (Chen et al., 2022). We present a recipe for highly efficient and stable training of a 22B-parameter ViT (ViT-22B) and perform a wide variety of experiments on the resulting model. When evaluated on downstream tasks (often with a lightweight linear model on frozen features), ViT-22B demonstrates increasing performance with scale. We further observe other interesting benefits of scale, including an improved tradeoff between fairness and performance, state-of-the-art alignment to human visual perception in terms of shape/texture bias, and improved robustness. ViT-22B demonstrates the potential for""LLM-like""scaling in vision, and provides key steps towards getting there." +87c5b281fa43e6f27191b20a8dd694eda1126336,2205.14135,10.52202/068431-1189,2022,FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness,Tri Dao,5,Neural Information Processing Systems,5706,"Transformers are slow and memory-hungry on long sequences, since the time and memory complexity of self-attention are quadratic in sequence length. Approximate attention methods have attempted to address this problem by trading off model quality to reduce the compute complexity, but often do not achieve wall-clock speedup. We argue that a missing principle is making attention algorithms IO-aware -- accounting for reads and writes between levels of GPU memory. We propose FlashAttention, an IO-aware exact attention algorithm that uses tiling to reduce the number of memory reads/writes between GPU high bandwidth memory (HBM) and GPU on-chip SRAM. We analyze the IO complexity of FlashAttention, showing that it requires fewer HBM accesses than standard attention, and is optimal for a range of SRAM sizes. We also extend FlashAttention to block-sparse attention, yielding an approximate attention algorithm that is faster than any existing approximate attention method. FlashAttention trains Transformers faster than existing baselines: 15% end-to-end wall-clock speedup on BERT-large (seq. length 512) compared to the MLPerf 1.1 training speed record, 3$\times$ speedup on GPT-2 (seq. length 1K), and 2.4$\times$ speedup on long-range arena (seq. length 1K-4K). FlashAttention and block-sparse FlashAttention enable longer context in Transformers, yielding higher quality models (0.7 better perplexity on GPT-2 and 6.4 points of lift on long-document classification) and entirely new capabilities: the first Transformers to achieve better-than-chance performance on the Path-X challenge (seq. length 16K, 61.4% accuracy) and Path-256 (seq. length 64K, 63.1% accuracy)." +094ff971d6a8b8ff870946c9b3ce5aa173617bfb,2204.02311,,2022,PaLM: Scaling Language Modeling with Pathways,A. Chowdhery,67,Journal of machine learning research,8533,"Large language models have been shown to achieve remarkable performance across a variety of natural language tasks using few-shot learning, which drastically reduces the number of task-specific training examples needed to adapt the model to a particular application. To further our understanding of the impact of scale on few-shot learning, we trained a 540-billion parameter, densely activated, Transformer language model, which we call Pathways Language Model PaLM. We trained PaLM on 6144 TPU v4 chips using Pathways, a new ML system which enables highly efficient training across multiple TPU Pods. We demonstrate continued benefits of scaling by achieving state-of-the-art few-shot learning results on hundreds of language understanding and generation benchmarks. On a number of these tasks, PaLM 540B achieves breakthrough performance, outperforming the finetuned state-of-the-art on a suite of multi-step reasoning tasks, and outperforming average human performance on the recently released BIG-bench benchmark. A significant number of BIG-bench tasks showed discontinuous improvements from model scale, meaning that performance steeply increased as we scaled to our largest model. PaLM also has strong capabilities in multilingual tasks and source code generation, which we demonstrate on a wide array of benchmarks. We additionally provide a comprehensive analysis on bias and toxicity, and study the extent of training data memorization with respect to model scale. Finally, we discuss the ethical considerations related to large language models and discuss potential mitigation strategies." +66c10bf1f11bc1b2d92204d8f8391d087f6de1c4,2104.09864,10.1016/j.neucom.2023.127063,2021,RoFormer: Enhanced Transformer with Rotary Position Embedding,Jian-Lin Su,6,Neurocomputing,6996,"Position encoding recently has shown effective in the transformer architecture. It enables valuable supervision for dependency modeling between elements at different positions of the sequence. In this paper, we first investigate various methods to integrate positional information into the learning process of transformer-based language models. Then, we propose a novel method named Rotary Position Embedding(RoPE) to effectively leverage the positional information. Specifically, the proposed RoPE encodes the absolute position with a rotation matrix and meanwhile incorporates the explicit relative position dependency in self-attention formulation. Notably, RoPE enables valuable properties, including the flexibility of sequence length, decaying inter-token dependency with increasing relative distances, and the capability of equipping the linear self-attention with relative position encoding. Finally, we evaluate the enhanced transformer with rotary position embedding, also called RoFormer, on various long text classification benchmark datasets. Our experiments show that it consistently overcomes its alternatives. Furthermore, we provide a theoretical analysis to explain some experimental results. RoFormer is already integrated into Huggingface: \url{https://huggingface.co/docs/transformers/model_doc/roformer}." +806adbb35ed4a95f51518f5962fd59685ad4706b,2010.04245,10.18653/v1/2020.findings-emnlp.379,2020,Query-Key Normalization for Transformers,Alex Henry,4,Findings,351,"Low-resource language translation is a challenging but socially valuable NLP task. Building on recent work adapting the Transformer’s normalization to this setting, we propose QKNorm, a normalization technique that modifies the attention mechanism to make the softmax function less prone to arbitrary saturation without sacrificing expressivity. Specifically, we apply l2-normalization along the head dimension of each query and key matrix prior to multiplying them and then scale up by a learnable parameter instead of dividing by the square root of the embedding dimension. We show improvements averaging 0.928 BLEU over state-of-the-art bilingual benchmarks for 5 low-resource translation pairs from the TED Talks corpus and IWSLT’15." +bdbf780dfd6b3eb0c9e980887feae5f23af15bc4,2002.05202,,2020,GLU Variants Improve Transformer,Noam Shazeer,1,arXiv.org,2410,"Gated Linear Units (arXiv:1612.08083) consist of the component-wise product of two linear projections, one of which is first passed through a sigmoid function. Variations on GLU are possible, using different nonlinear (or even linear) functions in place of sigmoid. We test these variants in the feed-forward sublayers of the Transformer (arXiv:1706.03762) sequence-to-sequence model, and find that some of them yield quality improvements over the typically-used ReLU or GELU activations." +b022f2a277a4bf5f42382e86e4380b96340b9e86,1608.03983,,2016,SGDR: Stochastic Gradient Descent with Warm Restarts,I. Loshchilov,2,International Conference on Learning Representations,11715,"Restart techniques are common in gradient-free optimization to deal with multimodal functions. Partial warm restarts are also gaining popularity in gradient-based optimization to improve the rate of convergence in accelerated gradient schemes to deal with ill-conditioned functions. In this paper, we propose a simple warm restart technique for stochastic gradient descent to improve its anytime performance when training deep neural networks. We empirically study its performance on the CIFAR-10 and CIFAR-100 datasets, where we demonstrate new state-of-the-art results at 3.14% and 16.21%, respectively. We also demonstrate its advantages on a dataset of EEG recordings and on a downsampled version of the ImageNet dataset. Our source code is available at this https URL" +a6b6eedb0559cb44bd3fcab64151529019bae42a,,,2021,Tuning Large Neural Networks via Zero-Shot Hyperparameter Transfer,Ge-feng Yang,10,Neural Information Processing Systems,221, +,,,,"speculative text. NO samples qualify, return an EMPTY LINE. Output ONLY the numbers of qualifying samples as a comma-separated list",,0,,0, +,,,,: Decay-free learning rate schedule via checkpoint merging,,0,Wsm,0, +,,,,: Optimal matrix sign methods and their application to the muon,,0,The polar express,0, +,,,,Gemma,,0,ai.google.dev/gemma/docs/ core,0, +,,,,"sample_efficient_gpt: Training framework for sample-efficient llm pretraining, 2026",,0,github,0, +,,,,Pleias,,0,. Trained on SYNTH dataset,0, diff --git a/02-literature/search/results/snowball-back-stability2024.csv b/02-literature/search/results/snowball-back-stability2024.csv new file mode 100644 index 0000000..13aec6e --- /dev/null +++ b/02-literature/search/results/snowball-back-stability2024.csv @@ -0,0 +1,22 @@ +s2_id,arxiv,doi,year,title,first_author,n_authors,venue,citations,abstract +5e969b6c033a3ec1665fc16296110d76be357f80,2406.17837,10.48550/arXiv.2406.17837,2024,Transformer Normalisation Layers and the Independence of Semantic Subspaces,Stephen Menary,3,arXiv.org,7,"Recent works have shown that transformers can solve contextual reasoning tasks by internally executing computational graphs called circuits. Circuits often use attention to logically match information from subspaces of the representation, e.g. using position-in-sequence to identify the previous token. In this work, we consider a semantic subspace to be any independent subspace of the latent representation that can fully determine an attention distribution. We show that Pre-Norm, the placement of normalisation layer used by state-of-the-art transformers, violates this ability unless the model learns a strict representation structure of orthogonal spheres. This is because it causes linear subspaces to interfere through their common normalisation factor. Theoretically, we analyse circuit stability by modelling this interference as random noise on the $L_2$-norms of the query/key/value vectors, predicting a phenomenon of circuit collapse when sparse-attention shifts to a different token. Empirically, we investigate the sensitivity of real-world models trained for mathematical addition, observing a 1% rate of circuit collapse when the norms are artificially perturbed by $\lesssim$10%. We contrast Pre-Norm with QKV-Norm, which places normalisation after the attention head's linear operators. Theoretically this relaxes the representational constraints. Empirically we observe comparable in-distribution but worse out-of-distribution performance." +f5789596531fad358c3166fdb5bd72d8e661c32c,2309.14322,10.48550/arXiv.2309.14322,2023,Small-scale proxies for large-scale Transformer training instabilities,Mitchell Wortsman,16,International Conference on Learning Representations,210,"Teams that have trained large Transformer-based models have reported training instabilities at large scale that did not appear when training with the same hyperparameters at smaller scales. Although the causes of such instabilities are of scientific interest, the amount of resources required to reproduce them has made investigation difficult. In this work, we seek ways to reproduce and study training stability and instability at smaller scales. First, we focus on two sources of training instability described in previous work: the growth of logits in attention layers (Dehghani et al., 2023) and divergence of the output logits from the log probabilities (Chowdhery et al., 2022). By measuring the relationship between learning rate and loss across scales, we show that these instabilities also appear in small models when training at high learning rates, and that mitigations previously employed at large scales are equally effective in this regime. This prompts us to investigate the extent to which other known optimizer and model interventions influence the sensitivity of the final loss to changes in the learning rate. To this end, we study methods such as warm-up, weight decay, and the $\mu$Param (Yang et al., 2022), and combine techniques to train small models that achieve similar losses across orders of magnitude of learning rate variation. Finally, to conclude our exploration we study two cases where instabilities can be predicted before they emerge by examining the scaling behavior of model activation and gradient norms." +d193675b92fbfbf22ed82fda35cd2e73587e33bd,2306.12929,10.48550/arXiv.2306.12929,2023,Quantizable Transformers: Removing Outliers by Helping Attention Heads Do Nothing,Yelysei Bondarenko,3,Neural Information Processing Systems,201,"Transformer models have been widely adopted in various domains over the last years, and especially large language models have advanced the field of AI significantly. Due to their size, the capability of these networks has increased tremendously, but this has come at the cost of a significant increase in necessary compute. Quantization is one of the most effective ways to reduce the computational time and memory consumption of neural networks. Many studies have shown, however, that modern transformer models tend to learn strong outliers in their activations, making them difficult to quantize. To retain acceptable performance, the existence of these outliers requires activations to be in higher bitwidth or the use of different numeric formats, extra fine-tuning, or other workarounds. We show that strong outliers are related to very specific behavior of attention heads that try to learn a""no-op""or just a partial update of the residual. To achieve the exact zeros needed in the attention matrix for a no-update, the input to the softmax is pushed to be larger and larger during training, causing outliers in other parts of the network. Based on these observations, we propose two simple (independent) modifications to the attention mechanism - clipped softmax and gated attention. We empirically show that models pre-trained using our methods learn significantly smaller outliers while maintaining and sometimes even improving the floating-point task performance. This enables us to quantize transformers to full INT8 quantization of the activations without any additional effort. We demonstrate the effectiveness of our methods on both language models (BERT, OPT) and vision transformers." +b3c64a046d6cfc0f55a2aebc5176bbab69a7e021,2304.13013,10.48550/arXiv.2304.13013,2023,Stable and low-precision training for large-scale vision-language models,Mitchell Wortsman,6,Neural Information Processing Systems,106,"We introduce new methods for 1) accelerating and 2) stabilizing training for large language-vision models. 1) For acceleration, we introduce SwitchBack, a linear layer for int8 quantized training which provides a speed-up of 13-25% while matching the performance of bfloat16 training within 0.1 percentage points for the 1B parameter CLIP ViT-Huge -- the largest int8 training to date. Our main focus is int8 as GPU support for float8 is rare, though we also analyze float8 training through simulation. While SwitchBack proves effective for float8, we show that standard techniques are also successful if the network is trained and initialized so that large feature magnitudes are discouraged, which we accomplish via layer-scale initialized with zeros. 2) For stability, we analyze loss spikes and find they consistently occur 1-8 iterations after the squared gradients become under-estimated by their AdamW second moment estimator. As a result, we recommend an AdamW-Adafactor hybrid which avoids loss spikes when training a CLIP ViT-Huge model and outperforms gradient clipping at the scales we test." +88c94b11bd18161d027a28dd758b58698063e029,2304.09871,10.48550/arXiv.2304.09871,2023,A Theory on Adam Instability in Large-Scale Machine Learning,Igor Molybog,17,arXiv.org,60,"We present a theory for the previously unexplained divergent behavior noticed in the training of large language models. We argue that the phenomenon is an artifact of the dominant optimization algorithm used for training, called Adam. We observe that Adam can enter a state in which the parameter update vector has a relatively large norm and is essentially uncorrelated with the direction of descent on the training loss landscape, leading to divergence. This artifact is more likely to be observed in the training of a deep model with a large batch size, which is the typical setting of large-scale language model training. To argue the theory, we present observations from the training runs of the language models of different scales: 7 billion, 30 billion, 65 billion, and 546 billion parameters." +385c363ea8e450f362d389f401beaeb5b42a0022,2303.06296,10.48550/arXiv.2303.06296,2023,Stabilizing Transformer Training by Preventing Attention Entropy Collapse,Shuangfei Zhai,8,International Conference on Machine Learning,221,"Training stability is of great importance to Transformers. In this work, we investigate the training dynamics of Transformers by examining the evolution of the attention layers. In particular, we track the attention entropy for each attention head during the course of training, which is a proxy for model sharpness. We identify a common pattern across different architectures and tasks, where low attention entropy is accompanied by high training instability, which can take the form of oscillating loss or divergence. We denote the pathologically low attention entropy, corresponding to highly concentrated attention scores, as $\textit{entropy collapse}$. As a remedy, we propose $\sigma$Reparam, a simple and efficient solution where we reparametrize all linear layers with spectral normalization and an additional learned scalar. We demonstrate that $\sigma$Reparam successfully prevents entropy collapse in the attention layers, promoting more stable training. Additionally, we prove a tight lower bound of the attention entropy, which decreases exponentially fast with the spectral norm of the attention logits, providing additional motivation for our approach. We conduct experiments with $\sigma$Reparam on image classification, image self-supervised learning, machine translation, speech recognition, and language modeling tasks. We show that $\sigma$Reparam provides stability and robustness with respect to the choice of hyperparameters, going so far as enabling training (a) a Vision Transformer {to competitive performance} without warmup, weight decay, layer normalization or adaptive optimizers; (b) deep architectures in machine translation and (c) speech recognition to competitive performance without warmup and adaptive optimizers. Code is available at \url{https://github.com/apple/ml-sigma-reparam}." +61e721334296ebfbbf6443b5ed9eb8c83b708c95,2302.05442,10.48550/arXiv.2302.05442,2023,Scaling Vision Transformers to 22 Billion Parameters,Mostafa Dehghani,42,International Conference on Machine Learning,972,"The scaling of Transformers has driven breakthrough capabilities for language models. At present, the largest large language models (LLMs) contain upwards of 100B parameters. Vision Transformers (ViT) have introduced the same architecture to image and video modelling, but these have not yet been successfully scaled to nearly the same degree; the largest dense ViT contains 4B parameters (Chen et al., 2022). We present a recipe for highly efficient and stable training of a 22B-parameter ViT (ViT-22B) and perform a wide variety of experiments on the resulting model. When evaluated on downstream tasks (often with a lightweight linear model on frozen features), ViT-22B demonstrates increasing performance with scale. We further observe other interesting benefits of scale, including an improved tradeoff between fairness and performance, state-of-the-art alignment to human visual perception in terms of shape/texture bias, and improved robustness. ViT-22B demonstrates the potential for""LLM-like""scaling in vision, and provides key steps towards getting there." +4a8964ea0de47010fb458021b68fa3ef5c4b77b2,2109.08668,,2021,Primer: Searching for Efficient Transformers for Language Modeling,David R. So,6,Neural Information Processing Systems,237,"Large Transformer models have been central to recent advances in natural language processing. The training and inference costs of these models, however, have grown rapidly and become prohibitively expensive. Here we aim to reduce the costs of Transformers by searching for a more efficient variant. Compared to previous approaches, our search is performed at a lower level, over the primitives that define a Transformer TensorFlow program. We identify an architecture, named Primer, that has a smaller training cost than the original Transformer and other variants for auto-regressive language modeling. Primer's improvements can be mostly attributed to two simple modifications: squaring ReLU activations and adding a depthwise convolution layer after each Q, K, and V projection in self-attention. Experiments show Primer's gains over Transformer increase as compute scale grows and follow a power law with respect to quality at optimal model sizes. We also verify empirically that Primer can be dropped into different codebases to significantly speed up training without additional tuning. For example, at a 500M parameter size, Primer improves the original T5 architecture on C4 auto-regressive language modeling, reducing the training cost by 4X. Furthermore, the reduced training cost means Primer needs much less compute to reach a target one-shot performance. For instance, in a 1.9B parameter configuration similar to GPT-3 XL, Primer uses 1/3 of the training compute to achieve the same one-shot performance as Transformer. We open source our models and several comparisons in T5 to help with reproducibility." +66c10bf1f11bc1b2d92204d8f8391d087f6de1c4,2104.09864,10.1016/j.neucom.2023.127063,2021,RoFormer: Enhanced Transformer with Rotary Position Embedding,Jian-Lin Su,6,Neurocomputing,6996,"Position encoding recently has shown effective in the transformer architecture. It enables valuable supervision for dependency modeling between elements at different positions of the sequence. In this paper, we first investigate various methods to integrate positional information into the learning process of transformer-based language models. Then, we propose a novel method named Rotary Position Embedding(RoPE) to effectively leverage the positional information. Specifically, the proposed RoPE encodes the absolute position with a rotation matrix and meanwhile incorporates the explicit relative position dependency in self-attention formulation. Notably, RoPE enables valuable properties, including the flexibility of sequence length, decaying inter-token dependency with increasing relative distances, and the capability of equipping the linear self-attention with relative position encoding. Finally, we evaluate the enhanced transformer with rotary position embedding, also called RoFormer, on various long text classification benchmark datasets. Our experiments show that it consistently overcomes its alternatives. Furthermore, we provide a theoretical analysis to explain some experimental results. RoFormer is already integrated into Huggingface: \url{https://huggingface.co/docs/transformers/model_doc/roformer}." +b364cdb02d18b9d9a3c097f5ea446f7e9ab10325,2103.17239,10.1109/ICCV48922.2021.00010,2021,Going deeper with Image Transformers,Hugo Touvron,5,IEEE International Conference on Computer Vision,1453,"Transformers have been recently adapted for large scale image classification, achieving high scores shaking up the long supremacy of convolutional neural networks. However the optimization of vision transformers has been little studied so far. In this work, we build and optimize deeper transformer networks for image classification. In particular, we investigate the interplay of architecture and optimization of such dedicated transformers. We make two architecture changes that significantly improve the accuracy of deep transformers. This leads us to produce models whose performance does not saturate early with more depth, for in-stance we obtain 86.5% top-1 accuracy on Imagenet when training with no external data, we thus attain the current sate of the art with less floating-point operations and parameters. Our best model establishes the new state of the art on Imagenet with Reassessed labels and Imagenet-V2 / match frequency, in the setting with no additional training data. We share our code and models1." +806adbb35ed4a95f51518f5962fd59685ad4706b,2010.04245,10.18653/v1/2020.findings-emnlp.379,2020,Query-Key Normalization for Transformers,Alex Henry,4,Findings,351,"Low-resource language translation is a challenging but socially valuable NLP task. Building on recent work adapting the Transformer’s normalization to this setting, we propose QKNorm, a normalization technique that modifies the attention mechanism to make the softmax function less prone to arbitrary saturation without sacrificing expressivity. Specifically, we apply l2-normalization along the head dimension of each query and key matrix prior to multiplying them and then scale up by a learnable parameter instead of dividing by the square root of the embedding dimension. We show improvements averaging 0.928 BLEU over state-of-the-art bilingual benchmarks for 5 low-resource translation pairs from the TED Talks corpus and IWSLT’15." +8323c591e119eb09b28b29fd6c7bc76bd889df7a,1909.08053,,2019,Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism,M. Shoeybi,6,arXiv.org,3332,"Recent work in language modeling demonstrates that training large transformer models advances the state of the art in Natural Language Processing applications. However, very large models can be quite difficult to train due to memory constraints. In this work, we present our techniques for training very large transformer models and implement a simple, efficient intra-layer model parallel approach that enables training transformer models with billions of parameters. Our approach does not require a new compiler or library changes, is orthogonal and complimentary to pipeline model parallelism, and can be fully implemented with the insertion of a few communication operations in native PyTorch. We illustrate this approach by converging transformer based models up to 8.3 billion parameters using 512 GPUs. We sustain 15.1 PetaFLOPs across the entire application with 76% scaling efficiency when compared to a strong single GPU baseline that sustains 39 TeraFLOPs, which is 30% of peak FLOPs. To demonstrate that large language models can further advance the state of the art (SOTA), we train an 8.3 billion parameter transformer language model similar to GPT-2 and a 3.9 billion parameter model similar to BERT. We show that careful attention to the placement of layer normalization in BERT-like models is critical to achieving increased performance as the model size grows. Using the GPT-2 model we achieve SOTA results on the WikiText103 (10.8 compared to SOTA perplexity of 15.8) and LAMBADA (66.5% compared to SOTA accuracy of 63.2%) datasets. Our BERT model achieves SOTA results on the RACE dataset (90.9% compared to SOTA accuracy of 89.4%)." +b5246fa284f86b544a7c31f050b3bd0defd053fd,1808.06226,10.18653/v1/D18-2012,2018,SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processing,Taku Kudo,2,Conference on Empirical Methods in Natural Language Processing,4390,"This paper describes SentencePiece, a language-independent subword tokenizer and detokenizer designed for Neural-based text processing, including Neural Machine Translation. It provides open-source C++ and Python implementations for subword units. While existing subword segmentation tools assume that the input is pre-tokenized into word sequences, SentencePiece can train subword models directly from raw sentences, which allows us to make a purely end-to-end and language independent system. We perform a validation experiment of NMT on English-Japanese machine translation, and find that it is possible to achieve comparable accuracy to direct subword training from raw sentences. We also compare the performance of subword training and segmentation with various configurations. SentencePiece is available under the Apache 2 license at https://github.com/google/sentencepiece." +d7878c2044fb699e0ce0cad83e411824b1499dc8,1611.09940,,2016,Neural Combinatorial Optimization with Reinforcement Learning,Irwan Bello,5,International Conference on Learning Representations,1946,"This paper presents a framework to tackle combinatorial optimization problems using neural networks and reinforcement learning. We focus on the traveling salesman problem (TSP) and train a recurrent network that, given a set of city coordinates, predicts a distribution over different city permutations. Using negative tour length as the reward signal, we optimize the parameters of the recurrent network using a policy gradient method. We compare learning the network parameters on a set of training graphs against learning them on individual test graphs. Despite the computational expense, without much engineering and heuristic designing, Neural Combinatorial Optimization achieves close to optimal results on 2D Euclidean graphs with up to 100 nodes. Applied to the KnapSack, another NP-hard problem, the same method obtains optimal solutions for instances with up to 200 items." +b022f2a277a4bf5f42382e86e4380b96340b9e86,1608.03983,,2016,SGDR: Stochastic Gradient Descent with Warm Restarts,I. Loshchilov,2,International Conference on Learning Representations,11715,"Restart techniques are common in gradient-free optimization to deal with multimodal functions. Partial warm restarts are also gaining popularity in gradient-based optimization to improve the rate of convergence in accelerated gradient schemes to deal with ill-conditioned functions. In this paper, we propose a simple warm restart technique for stochastic gradient descent to improve its anytime performance when training deep neural networks. We empirically study its performance on the CIFAR-10 and CIFAR-100 datasets, where we demonstrate new state-of-the-art results at 3.14% and 16.21%, respectively. We also demonstrate its advantages on a dataset of EEG recordings and on a downsampled version of the ImageNet dataset. Our source code is available at this https URL" +0c908739fbff75f03469d13d4a1a07de3414ee19,1503.02531,,2015,Distilling the Knowledge in a Neural Network,Geoffrey E. Hinton,3,arXiv.org,27151,"A very simple way to improve the performance of almost any machine learning algorithm is to train many different models on the same data and then to average their predictions. Unfortunately, making predictions using a whole ensemble of models is cumbersome and may be too computationally expensive to allow deployment to a large number of users, especially if the individual models are large neural nets. Caruana and his collaborators have shown that it is possible to compress the knowledge in an ensemble into a single model which is much easier to deploy and we develop this approach further using a different compression technique. We achieve some surprising results on MNIST and we show that we can significantly improve the acoustic model of a heavily used commercial system by distilling the knowledge in an ensemble of models into a single model. We also introduce a new type of ensemble composed of one or more full models and many specialist models which learn to distinguish fine-grained classes that the full models confuse. Unlike a mixture of experts, these specialist models can be trained rapidly and in parallel." +9405cc0d6169988371b2755e573cc28650d14dfe,,,2019,Language Models are Unsupervised Multitask Learners,A. Radford,6,,30929, +,,,2015,A method for stochastic optimization,,0,International Conference on Learning Representations (ICLR),0, +,,,,Scaling language modeling with pathways,,0,J. Mach. Learn. Res.,0, +,,,,pre-trained transformer,,0,Opt: Open,0, +,,,,Nvidia h100 tensor core gpu,,0,nvidia.,0, diff --git a/02-literature/search/results/snowball-back-vit22b.csv b/02-literature/search/results/snowball-back-vit22b.csv new file mode 100644 index 0000000..1221bd0 --- /dev/null +++ b/02-literature/search/results/snowball-back-vit22b.csv @@ -0,0 +1,135 @@ +s2_id,arxiv,doi,year,title,first_author,n_authors,venue,citations,abstract +3692f4df9d11af68f9b9c9a526667db3f99e552c,,10.1145/3567955.3567959,2022,Overlap Communication with Dependent Computation via Decomposition in Large Deep Learning Models,Shi-Bo Wang,14,International Conference on Architectural Support for Programming Languages and Operating Systems,132,"Large deep learning models have shown great potential with state-of-the-art results in many tasks. However, running these large models is quite challenging on an accelerator (GPU or TPU) because the on-device memory is too limited for the size of these models. Intra-layer model parallelism is an approach to address the issues by partitioning individual layers or operators across multiple devices in a distributed accelerator cluster. But, the data communications generated by intra-layer model parallelism can contribute to a significant proportion of the overall execution time and severely hurt the computational efficiency. As intra-layer model parallelism is critical to enable large deep learning models, this paper proposes a novel technique to effectively reduce its data communication overheads by overlapping communication with computation. With the proposed technique, an identified original communication collective is decomposed along with the dependent computation operation into a sequence of finer-grained operations. By creating more overlapping opportunities and executing the newly created, finer-grained communication and computation operations in parallel, it effectively hides the data transfer latency and achieves a better system utilization. Evaluated on TPU v4 Pods using different types of large models that have 10 billion to 1 trillion parameters, the proposed technique improves system throughput by 1.14 - 1.38x. The achieved highest peak FLOPS utilization is 72% on 1024 TPU chips with a large language model that has 500 billion parameters." +925fe4b2225e534888a2c78c9f6539a8e4e58d59,2212.08013,10.1109/CVPR52729.2023.01393,2022,FlexiViT: One Model for All Patch Sizes,Lucas Beyer,10,Computer Vision and Pattern Recognition,204,"Vision Transformers convert images to sequences by slicing them into patches. The size of these patches controls a speed/accuracy tradeoff, with smaller patches leading to higher accuracy at greater computational cost, but changing the patch size typically requires retraining the model. In this paper, we demonstrate that simply randomizing the patch size at training time leads to a single set of weights that performs well across a wide range of patch sizes, making it possible to tailor the model to different compute budgets at deployment time. We extensively evaluate the resulting model, which we call FlexiViT, on a wide range of tasks, including classification, image-text retrieval, open-world detection, panoptic segmentation, and semantic segmentation, concluding that it usually matches, and sometimes outperforms, standard ViT models trained at a single patch size in an otherwise identical setup. Hence, FlexiViT training is a simple drop-in improvement for ViT that makes it easy to add compute-adaptive capabilities to most models relying on a ViT backbone architecture. Code and pre-trained models are available at github.com/google-research/big_vision." +325d8e9501af05e594bd668b6cd6d43ed42c8b4d,2212.03191,10.48550/arXiv.2212.03191,2022,InternVideo: General Video Foundation Models via Generative and Discriminative Learning,Yi Wang,17,arXiv.org,589,"The foundation models have recently shown excellent performance on a variety of downstream tasks in computer vision. However, most existing vision foundation models simply focus on image-level pretraining and adpation, which are limited for dynamic and complex video-level understanding tasks. To fill the gap, we present general video foundation models, InternVideo, by taking advantage of both generative and discriminative self-supervised video learning. Specifically, InternVideo efficiently explores masked video modeling and video-language contrastive learning as the pretraining objectives, and selectively coordinates video representations of these two complementary frameworks in a learnable manner to boost various video applications. Without bells and whistles, InternVideo achieves state-of-the-art performance on 39 video datasets from extensive tasks including video action recognition/detection, video-language alignment, and open-world video applications. Especially, our methods can obtain 91.1% and 77.2% top-1 accuracy on the challenging Kinetics-400 and Something-Something V2 benchmarks, respectively. All of these results effectively show the generality of our InternVideo for video understanding. The code will be released at https://github.com/OpenGVLab/InternVideo ." +e3efdaa40ad4c9a604f98252fbc5231f923fedfd,2211.10193,10.48550/arXiv.2211.10193,2022,Layer-Stack Temperature Scaling,Amr Khalifa,5,arXiv.org,2,"Recent works demonstrate that early layers in a neural network contain useful information for prediction. Inspired by this, we show that extending temperature scaling across all layers improves both calibration and accuracy. We call this procedure""layer-stack temperature scaling""(LATES). Informally, LATES grants each layer a weighted vote during inference. We evaluate it on five popular convolutional neural network architectures both in- and out-of-distribution and observe a consistent improvement over temperature scaling in terms of accuracy, calibration, and AUC. All conclusions are supported by comprehensive statistical analyses. Since LATES neither retrains the architecture nor introduces many more parameters, its advantages can be reaped without requiring additional data beyond what is used in temperature scaling. Finally, we show that combining LATES with Monte Carlo Dropout matches state-of-the-art results on CIFAR10/100." +379e42895f6d40ab9e9559609f505aba89145a5d,2211.05102,10.48550/arXiv.2211.05102,2022,Efficiently Scaling Transformer Inference,Reiner Pope,10,Conference on Machine Learning and Systems,827,"We study the problem of efficient generative inference for Transformer models, in one of its most challenging settings: large deep models, with tight latency targets and long sequence lengths. Better understanding of the engineering tradeoffs for inference for large Transformer-based models is important as use cases of these models are growing rapidly throughout application areas. We develop a simple analytical model for inference efficiency to select the best multi-dimensional partitioning techniques optimized for TPU v4 slices based on the application requirements. We combine these with a suite of low-level optimizations to achieve a new Pareto frontier on the latency and model FLOPS utilization (MFU) tradeoffs on 500B+ parameter models that outperforms the FasterTransformer suite of benchmarks. We further show that with appropriate partitioning, the lower memory requirements of multiquery attention (i.e. multiple query heads share single key/value head) enables scaling up to 32x larger context lengths. Finally, we achieve a low-batch-size latency of 29ms per token during generation (using int8 weight quantization) and a 76% MFU during large-batch-size processing of input tokens, while supporting a long 2048-token context length on the PaLM 540B parameter model." +cdbd4f9b6ab2e2fd1ddf5400d5ed2c18960635d1,2210.11416,10.48550/arXiv.2210.11416,2022,Scaling Instruction-Finetuned Language Models,Hyung Won Chung,32,Journal of machine learning research,4540,"Finetuning language models on a collection of datasets phrased as instructions has been shown to improve model performance and generalization to unseen tasks. In this paper we explore instruction finetuning with a particular focus on (1) scaling the number of tasks, (2) scaling the model size, and (3) finetuning on chain-of-thought data. We find that instruction finetuning with the above aspects dramatically improves performance on a variety of model classes (PaLM, T5, U-PaLM), prompting setups (zero-shot, few-shot, CoT), and evaluation benchmarks (MMLU, BBH, TyDiQA, MGSM, open-ended generation). For instance, Flan-PaLM 540B instruction-finetuned on 1.8K tasks outperforms PALM 540B by a large margin (+9.4% on average). Flan-PaLM 540B achieves state-of-the-art performance on several benchmarks, such as 75.2% on five-shot MMLU. We also publicly release Flan-T5 checkpoints, which achieve strong few-shot performance even compared to much larger models, such as PaLM 62B. Overall, instruction finetuning is a general method for improving the performance and usability of pretrained language models." +28630034bb29760df01ab033b743e30b37f336ae,2209.06794,10.48550/arXiv.2209.06794,2022,PaLI: A Jointly-Scaled Multilingual Language-Image Model,Xi Chen,29,International Conference on Learning Representations,1055,"Effective scaling and a flexible task interface enable large language models to excel at many tasks. We present PaLI (Pathways Language and Image model), a model that extends this approach to the joint modeling of language and vision. PaLI generates text based on visual and textual inputs, and with this interface performs many vision, language, and multimodal tasks, in many languages. To train PaLI, we make use of large pre-trained encoder-decoder language models and Vision Transformers (ViTs). This allows us to capitalize on their existing capabilities and leverage the substantial cost of training them. We find that joint scaling of the vision and language components is important. Since existing Transformers for language are much larger than their vision counterparts, we train a large, 4-billion parameter ViT (ViT-e) to quantify the benefits from even larger-capacity vision models. To train PaLI, we create a large multilingual mix of pretraining tasks, based on a new image-text training set containing 10B images and texts in over 100 languages. PaLI achieves state-of-the-art in multiple vision and language tasks (such as captioning, visual question-answering, scene-text understanding), while retaining a simple, modular, and scalable design." +9da634823416e96417530f0a2197b9a4936eee3e,2207.07411,10.48550/arXiv.2207.07411,2022,Plex: Towards Reliability using Pretrained Large Model Extensions,Dustin Tran,26,arXiv.org,150,"A recent trend in artificial intelligence is the use of pretrained models for language and vision tasks, which have achieved extraordinary performance but also puzzling failures. Probing these models' abilities in diverse ways is therefore critical to the field. In this paper, we explore the reliability of models, where we define a reliable model as one that not only achieves strong predictive performance but also performs well consistently over many decision-making tasks involving uncertainty (e.g., selective prediction, open set recognition), robust generalization (e.g., accuracy and proper scoring rules such as log-likelihood on in- and out-of-distribution datasets), and adaptation (e.g., active learning, few-shot uncertainty). We devise 10 types of tasks over 40 datasets in order to evaluate different aspects of reliability on both vision and language domains. To improve reliability, we developed ViT-Plex and T5-Plex, pretrained large model extensions for vision and language modalities, respectively. Plex greatly improves the state-of-the-art across reliability tasks, and simplifies the traditional protocol as it improves the out-of-the-box performance and does not require designing scores or tuning the model for each task. We demonstrate scaling effects over model sizes up to 1B parameters and pretraining dataset sizes up to 4B examples. We also demonstrate Plex's capabilities on challenging tasks including zero-shot open set recognition, active learning, and uncertainty in conversational language understanding." +1243e13254bb4ea1f71b4be8a3e4e54ffd02d2fe,2206.10789,10.48550/arXiv.2206.10789,2022,Scaling Autoregressive Models for Content-Rich Text-to-Image Generation,Jia-Hui Yu,17,Trans. Mach. Learn. Res.,1591,"We present the Pathways Autoregressive Text-to-Image (Parti) model, which generates high-fidelity photorealistic images and supports content-rich synthesis involving complex compositions and world knowledge. Parti treats text-to-image generation as a sequence-to-sequence modeling problem, akin to machine translation, with sequences of image tokens as the target outputs rather than text tokens in another language. This strategy can naturally tap into the rich body of prior work on large language models, which have seen continued advances in capabilities and performance through scaling data and model sizes. Our approach is simple: First, Parti uses a Transformer-based image tokenizer, ViT-VQGAN, to encode images as sequences of discrete tokens. Second, we achieve consistent quality improvements by scaling the encoder-decoder Transformer model up to 20B parameters, with a new state-of-the-art zero-shot FID score of 7.23 and finetuned FID score of 3.22 on MS-COCO. Our detailed analysis on Localized Narratives as well as PartiPrompts (P2), a new holistic benchmark of over 1600 English prompts, demonstrate the effectiveness of Parti across a wide variety of categories and difficulty aspects. We also explore and highlight limitations of our models in order to define and exemplify key areas of focus for further improvements. See https://parti.research.google/ for high-resolution images." +dac3a172b504f4e33c029655e9befb3386e5f63a,2206.07682,10.48550/arXiv.2206.07682,2022,Emergent Abilities of Large Language Models,Jason Wei,16,Trans. Mach. Learn. Res.,3931,"Scaling up language models has been shown to predictably improve performance and sample efficiency on a wide range of downstream tasks. This paper instead discusses an unpredictable phenomenon that we refer to as emergent abilities of large language models. We consider an ability to be emergent if it is not present in smaller models but is present in larger models. Thus, emergent abilities cannot be predicted simply by extrapolating the performance of smaller models. The existence of such emergence implies that additional scaling could further expand the range of capabilities of language models." +9695824d7a01fad57ba9c01d7d76a519d78d65e7,2205.11487,10.48550/arXiv.2205.11487,2022,Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding,Chitwan Saharia,14,Neural Information Processing Systems,9113,"We present Imagen, a text-to-image diffusion model with an unprecedented degree of photorealism and a deep level of language understanding. Imagen builds on the power of large transformer language models in understanding text and hinges on the strength of diffusion models in high-fidelity image generation. Our key discovery is that generic large language models (e.g. T5), pretrained on text-only corpora, are surprisingly effective at encoding text for image synthesis: increasing the size of the language model in Imagen boosts both sample fidelity and image-text alignment much more than increasing the size of the image diffusion model. Imagen achieves a new state-of-the-art FID score of 7.27 on the COCO dataset, without ever training on COCO, and human raters find Imagen samples to be on par with the COCO data itself in image-text alignment. To assess text-to-image models in greater depth, we introduce DrawBench, a comprehensive and challenging benchmark for text-to-image models. With DrawBench, we compare Imagen with recent methods including VQ-GAN+CLIP, Latent Diffusion Models, and DALL-E 2, and find that human raters prefer Imagen over other models in side-by-side comparisons, both in terms of sample quality and image-text alignment. See https://imagen.research.google/ for an overview of the results." +a26a7a74f1e5fd562be95c3611a0680759fbdf84,2205.01917,10.48550/arXiv.2205.01917,2022,CoCa: Contrastive Captioners are Image-Text Foundation Models,Jia-Hui Yu,6,Trans. Mach. Learn. Res.,1884,"Exploring large-scale pretrained foundation models is of significant interest in computer vision because these models can be quickly transferred to many downstream tasks. This paper presents Contrastive Captioner (CoCa), a minimalist design to pretrain an image-text encoder-decoder foundation model jointly with contrastive loss and captioning loss, thereby subsuming model capabilities from contrastive approaches like CLIP and generative methods like SimVLM. In contrast to standard encoder-decoder transformers where all decoder layers attend to encoder outputs, CoCa omits cross-attention in the first half of decoder layers to encode unimodal text representations, and cascades the remaining decoder layers which cross-attend to the image encoder for multimodal image-text representations. We apply a contrastive loss between unimodal image and text embeddings, in addition to a captioning loss on the multimodal decoder outputs which predicts text tokens autoregressively. By sharing the same computational graph, the two training objectives are computed efficiently with minimal overhead. CoCa is pretrained end-to-end and from scratch on both web-scale alt-text data and annotated images by treating all labels simply as text, seamlessly unifying natural language supervision for representation learning. Empirically, CoCa achieves state-of-the-art performance with zero-shot transfer or minimal task-specific adaptation on a broad range of downstream tasks, spanning visual recognition (ImageNet, Kinetics-400/600/700, Moments-in-Time), crossmodal retrieval (MSCOCO, Flickr30K, MSR-VTT), multimodal understanding (VQA, SNLI-VE, NLVR2), and image captioning (MSCOCO, NoCaps). Notably on ImageNet classification, CoCa obtains 86.3% zero-shot top-1 accuracy, 90.6% with a frozen encoder and learned classification head, and new state-of-the-art 91.0% top-1 accuracy on ImageNet with a finetuned encoder." +d2f63b56fc6bc373f5c023454c2b253326962865,2204.07118,10.48550/arXiv.2204.07118,2022,DeiT III: Revenge of the ViT,Hugo Touvron,3,European Conference on Computer Vision,729,"A Vision Transformer (ViT) is a simple neural architecture amenable to serve several computer vision tasks. It has limited built-in architectural priors, in contrast to more recent architectures that incorporate priors either about the input data or of specific tasks. Recent works show that ViTs benefit from self-supervised pre-training, in particular BerT-like pre-training like BeiT. In this paper, we revisit the supervised training of ViTs. Our procedure builds upon and simplifies a recipe introduced for training ResNet-50. It includes a new simple data-augmentation procedure with only 3 augmentations, closer to the practice in self-supervised learning. Our evaluations on Image classification (ImageNet-1k with and without pre-training on ImageNet-21k), transfer learning and semantic segmentation show that our procedure outperforms by a large margin previous fully supervised training recipes for ViT. It also reveals that the performance of our ViT trained with supervision is comparable to that of more recent architectures. Our results could serve as better baselines for recent self-supervised approaches demonstrated on ViT." +094ff971d6a8b8ff870946c9b3ce5aa173617bfb,2204.02311,,2022,PaLM: Scaling Language Modeling with Pathways,A. Chowdhery,67,Journal of machine learning research,8533,"Large language models have been shown to achieve remarkable performance across a variety of natural language tasks using few-shot learning, which drastically reduces the number of task-specific training examples needed to adapt the model to a particular application. To further our understanding of the impact of scale on few-shot learning, we trained a 540-billion parameter, densely activated, Transformer language model, which we call Pathways Language Model PaLM. We trained PaLM on 6144 TPU v4 chips using Pathways, a new ML system which enables highly efficient training across multiple TPU Pods. We demonstrate continued benefits of scaling by achieving state-of-the-art few-shot learning results on hundreds of language understanding and generation benchmarks. On a number of these tasks, PaLM 540B achieves breakthrough performance, outperforming the finetuned state-of-the-art on a suite of multi-step reasoning tasks, and outperforming average human performance on the recently released BIG-bench benchmark. A significant number of BIG-bench tasks showed discontinuous improvements from model scale, meaning that performance steeply increased as we scaled to our largest model. PaLM also has strong capabilities in multilingual tasks and source code generation, which we demonstrate on a wide array of benchmarks. We additionally provide a comprehensive analysis on bias and toxicity, and study the extent of training data memorization with respect to model scale. Finally, we discuss the ethical considerations related to large language models and discuss potential mitigation strategies." +74393ce60e64238d5ed0ff9952255d17b3be7092,2203.04946,,2022,Do better ImageNet classifiers assess perceptual similarity better?,Manoj Kumar,4,Trans. Mach. Learn. Res.,42,"Perceptual distances between images, as measured in the space of pre-trained deep features, have outperformed prior low-level, pixel-based metrics on assessing perceptual similarity. While the capabilities of older and less accurate models such as AlexNet and VGG to capture perceptual similarity are well known, modern and more accurate models are less studied. In this paper, we present a large-scale empirical study to assess how well ImageNet classifiers perform on perceptual similarity. First, we observe a inverse correlation between ImageNet accuracy and Perceptual Scores of modern networks such as ResNets, EfficientNets, and Vision Transformers: that is better classifiers achieve worse Perceptual Scores. Then, we examine the ImageNet accuracy/Perceptual Score relationship on varying the depth, width, number of training steps, weight decay, label smoothing, and dropout. Higher accuracy improves Perceptual Score up to a certain point, but we uncover a Pareto frontier between accuracies and Perceptual Score in the mid-to-high accuracy regime. We explore this relationship further using a number of plausible hypotheses such as distortion invariance, spatial frequency sensitivity, and alternative perceptual functions. Interestingly we discover shallow ResNets and ResNets trained for less than 5 epochs only on ImageNet, whose emergent Perceptual Score matches the prior best networks trained directly on supervised human perceptual judgements. The checkpoints for the models in our study are available at https://console.cloud.google.com/storage/browser/gresearch/perceptual_similarity." +08e6b161480249514fab4d6154bea3f5d6ddd4c6,2201.03529,,2022,Head2Toe: Utilizing Intermediate Representations for Better Transfer Learning,Utku Evci,4,International Conference on Machine Learning,113,"Transfer-learning methods aim to improve performance in a data-scarce target domain using a model pretrained on a data-rich source domain. A cost-efficient strategy, linear probing, involves freezing the source model and training a new classification head for the target domain. This strategy is outperformed by a more costly but state-of-the-art method -- fine-tuning all parameters of the source model to the target domain -- possibly because fine-tuning allows the model to leverage useful information from intermediate layers which is otherwise discarded by the later pretrained layers. We explore the hypothesis that these intermediate layers might be directly exploited. We propose a method, Head-to-Toe probing (Head2Toe), that selects features from all layers of the source model to train a classification head for the target-domain. In evaluations on the VTAB-1k, Head2Toe matches performance obtained with fine-tuning on average while reducing training and storage cost hundred folds or more, but critically, for out-of-distribution transfer, Head2Toe outperforms fine-tuning." +d257547d681f3a153f4bbf85f5955b5a0189e500,2111.10050,10.1016/j.neucom.2023.126658,2021,Combined Scaling for Zero-shot Transfer Learning,Hieu Pham,12,Neurocomputing,252,"We present a combined scaling method - named BASIC - that achieves 85.7% top-1 accuracy on the ImageNet ILSVRC-2012 validation set without learning from any labeled ImageNet example. This accuracy surpasses best published similar models - CLIP and ALIGN - by 9.3%. Our BASIC model also shows significant improvements in robustness benchmarks. For instance, on 5 test sets with natural distribution shifts such as ImageNet-{A,R,V2,Sketch} and ObjectNet, our model achieves 84.3% top-1 average accuracy, only a small drop from its original ImageNet accuracy. To achieve these results, we scale up the contrastive learning framework of CLIP and ALIGN in three dimensions: data size, model size, and batch size. Our dataset has 6.6B noisy image-text pairs, which is 4x larger than ALIGN, and 16x larger than CLIP. Our largest model has 3B weights, which is 3.75x larger in parameters and 8x larger in FLOPs than ALIGN and CLIP. Finally, our batch size is 65536 which is 2x more than CLIP and 4x more than ALIGN. We encountered two main challenges with the scaling rules of BASIC. First, the main challenge with implementing the combined scaling rules of BASIC is the limited memory of accelerators, such as GPUs and TPUs. To overcome the memory limit, we propose two simple methods which make use of gradient checkpointing and model parallelism. Second, while increasing the dataset size and the model size has been the defacto method to improve the performance of deep learning models like BASIC, the effect of a large contrastive batch size on such contrastive-trained image-text models is not well-understood. To shed light on the benefits of large contrastive batch sizes, we develop a theoretical framework which shows that larger contrastive batch sizes lead to smaller generalization gaps for image-text models such as BASIC." +197d5867a45a2988f4dd159063cdfbfe90164962,2111.07991,10.1109/CVPR52688.2022.01759,2021,LiT: Zero-Shot Transfer with Locked-image text Tuning,Xiao-Hua Zhai,7,Computer Vision and Pattern Recognition,821,"This paper presents contrastive-tuning, a simple method employing contrastive training to align image and text mod-els while still taking advantage of their pre-training. In our empirical study we find that locked pre-trained image mod-els with unlocked text models work best. We call this in-stance of contrastive-tuning “Locked-image Tuning” (LiT), which just teaches a text model to read out good repre-sentations from a pre-trained image model for new tasks. A LiT model gains the capability of zero-shot transfer to new vision tasks, such as image classification or retrieval. The proposed LiT is widely applicable; it works reliably with multiple pre-training methods (supervised and unsu-pervised) and across diverse architectures (ResNet, Vision Transformers and MLP-Mixer) using three different image-text datasets. With the transformer-based pre-trained ViT-g/14 model, the LiT model achieves 84.5% zero-shot trans-fer accuracy on the ImageNet test set, and 81.1% on the challenging out-of-distribution ObjectNet test set." +66d735987a31d666a6459566ae026c40ab9a1c3a,2110.12894,,2021,The Efficiency Misnomer,Mostafa Dehghani,5,International Conference on Learning Representations,130,"Model efficiency is a critical aspect of developing and deploying machine learning models. Inference time and latency directly affect the user experience, and some applications have hard requirements. In addition to inference costs, model training also have direct financial and environmental impacts. Although there are numerous well-established metrics (cost indicators) for measuring model efficiency, researchers and practitioners often assume that these metrics are correlated with each other and report only few of them. In this paper, we thoroughly discuss common cost indicators, their advantages and disadvantages, and how they can contradict each other. We demonstrate how incomplete reporting of cost indicators can lead to partial conclusions and a blurred or incomplete picture of the practical considerations of different models. We further present suggestions to improve reporting of efficiency metrics." +b8cb9c0b02da96a9908665ae67692a6da4dd25a4,2110.11403,10.1109/CVPR52688.2022.02070,2021,SCENIC: A JAX Library for Computer Vision Research and Beyond,Mostafa Dehghani,5,Computer Vision and Pattern Recognition,79,"Scenic is an open-source11https://github.com/google-research/scenic JAX library with a focus on transformer-based models for computer vision research and beyond. The goal of this toolkit is to facilitate rapid experimentation, prototyping, and research of new architectures and models. Scenic supports a diverse range of tasks (e.g., classification, segmentation, detection) and facilitates working on multi-modal problems, along with GPU/TPU support for large-scale, multi-host and multi-device training. Scenic also offers optimized implementations of state-of-the-art research models spanning a wide range of modalities. Scenic has been successfully used for numerous projects and published papers and continues serving as the library of choice for rapid prototyping and publication of new research ideas." +c206a6e7f51f5e1b6bfc479a174b66ad88ada2db,2110.02095,,2021,Exploring the Limits of Large Scale Pre-training,Samira Abnar,4,International Conference on Learning Representations,141,"Recent developments in large-scale machine learning suggest that by scaling up data, model size and training time properly, one might observe that improvements in pre-training would transfer favorably to most downstream tasks. In this work, we systematically study this phenomena and establish that, as we increase the upstream accuracy, the performance of downstream tasks saturates. In particular, we investigate more than 4800 experiments on Vision Transformers, MLP-Mixers and ResNets with number of parameters ranging from ten million to ten billion, trained on the largest scale of available image data (JFT, ImageNet21K) and evaluated on more than 20 downstream image recognition tasks. We propose a model for downstream performance that reflects the saturation phenomena and captures the nonlinear relationship in performance of upstream and downstream tasks. Delving deeper to understand the reasons that give rise to these phenomena, we show that the saturation behavior we observe is closely related to the way that representations evolve through the layers of the models. We showcase an even more extreme scenario where performance on upstream and downstream are at odds with each other. That is, to have a better downstream performance, we need to hurt upstream accuracy." +e7735be4f0fc427515fd7206ebc714356b97e71a,2107.07002,,2021,The Benchmark Lottery,Mostafa Dehghani,8,arXiv.org,183,"The world of empirical machine learning (ML) strongly relies on benchmarks in order to determine the relative effectiveness of different algorithms and methods. This paper proposes the notion of""a benchmark lottery""that describes the overall fragility of the ML benchmarking process. The benchmark lottery postulates that many factors, other than fundamental algorithmic superiority, may lead to a method being perceived as superior. On multiple benchmark setups that are prevalent in the ML community, we show that the relative performance of algorithms may be altered significantly simply by choosing different benchmark tasks, highlighting the fragility of the current paradigms and potential fallacious interpretation derived from benchmarking ML methods. Given that every benchmark makes a statement about what it perceives to be important, we argue that this might lead to biased progress in the community. We discuss the implications of the observed phenomena and provide recommendations on mitigating them using multiple machine learning domains and communities as use cases, including natural language processing, computer vision, information retrieval, recommender systems, and reinforcement learning." +6034e7ec13f1bd38f67cc4cd0f588176447571ca,2107.04212,10.18653/v1/2021.woah-1.5,2021,Measuring and Improving Model-Moderator Collaboration using Uncertainty Estimation,Ian Kivlichan,4,WOAH,28,"Content moderation is often performed by a collaboration between humans and machine learning models. However, it is not well understood how to design the collaborative process so as to maximize the combined moderator-model system performance. This work presents a rigorous study of this problem, focusing on an approach that incorporates model uncertainty into the collaborative process. First, we introduce principled metrics to describe the performance of the collaborative system under capacity constraints on the human moderator, quantifying how efficiently the combined system utilizes human decisions. Using these metrics, we conduct a large benchmark study evaluating the performance of state-of-the-art uncertainty models under different collaborative review strategies. We find that an uncertainty-based strategy consistently outperforms the widely used strategy based on toxicity scores, and moreover that the choice of review strategy drastically changes the overall system performance. Our results demonstrate the importance of rigorous metrics for understanding and developing effective moderator-model systems for content moderation, as well as the utility of uncertainty estimation in this domain." +4b1db6ebbdfcfe8ef67c5db511b6ad169fcc8f7f,2106.15831,,2021,The Evolution of Out-of-Distribution Robustness Throughout Fine-Tuning,Anders Andreassen,4,Trans. Mach. Learn. Res.,97,"Although machine learning models typically experience a drop in performance on out-of-distribution data, accuracies on in- versus out-of-distribution data are widely observed to follow a single linear trend when evaluated across a testbed of models. Models that are more accurate on the out-of-distribution data relative to this baseline exhibit""effective robustness""and are exceedingly rare. Identifying such models, and understanding their properties, is key to improving out-of-distribution performance. We conduct a thorough empirical investigation of effective robustness during fine-tuning and surprisingly find that models pre-trained on larger datasets exhibit effective robustness during training that vanishes at convergence. We study how properties of the data influence effective robustness, and we show that it increases with the larger size, more diversity, and higher example difficulty of the dataset. We also find that models that display effective robustness are able to correctly classify 10% of the examples that no other current testbed model gets correct. Finally, we discuss several strategies for scaling effective robustness to the high-accuracy regime to improve the out-of-distribution accuracy of state-of-the-art models." +e757488d2e8684e3da7b14fbb000b7e4a0bab001,2106.07998,,2021,Revisiting the Calibration of Modern Neural Networks,M. Minderer,8,Neural Information Processing Systems,654,"Accurate estimation of predictive uncertainty (model calibration) is essential for the safe application of neural networks. Many instances of miscalibration in modern neural networks have been reported, suggesting a trend that newer, more accurate models produce poorly calibrated predictions. Here, we revisit this question for recent state-of-the-art image classification models. We systematically relate model calibration and accuracy, and find that the most recent models, notably those not using convolutions, are among the best calibrated. Trends observed in prior model generations, such as decay of calibration with distribution shift or model size, are less pronounced in recent architectures. We also show that model size and amount of pretraining do not fully explain these differences, suggesting that architecture is a major determinant of calibration properties." +0e100c06d9fbdf32c434fd40469939a4aaab6c24,2106.07411,,2021,Partial success in closing the gap between human and machine vision,Robert Geirhos,7,Neural Information Processing Systems,306,"A few years ago, the first CNN surpassed human performance on ImageNet. However, it soon became clear that machines lack robustness on more challenging test cases, a major obstacle towards deploying machines""in the wild""and towards obtaining better computational models of human visual perception. Here we ask: Are we making progress in closing the gap between human and machine vision? To answer this question, we tested human observers on a broad range of out-of-distribution (OOD) datasets, recording 85,120 psychophysical trials across 90 participants. We then investigated a range of promising machine learning developments that crucially deviate from standard supervised CNNs along three axes: objective function (self-supervised, adversarially trained, CLIP language-image training), architecture (e.g. vision transformers), and dataset size (ranging from 1M to 1B). Our findings are threefold. (1.) The longstanding distortion robustness gap between humans and CNNs is closing, with the best models now exceeding human feedforward performance on most of the investigated OOD datasets. (2.) There is still a substantial image-level consistency gap, meaning that humans make different errors than models. In contrast, most models systematically agree in their categorisation errors, even substantially different ones like contrastive self-supervised vs. standard supervised models. (3.) In many cases, human-to-model consistency improves when training dataset size is increased by one to three orders of magnitude. Our results give reason for cautious optimism: While there is still much room for improvement, the behavioural difference between human and machine vision is narrowing. In order to measure future progress, 17 OOD datasets with image-level human behavioural data and evaluation code are provided as a toolbox and benchmark at: https://github.com/bethgelab/model-vs-human/" +8690d62d4bbbd0b1ed5e1f25320d10853bfbeb01,2106.05974,,2021,Scaling Vision with Sparse Mixture of Experts,C. Riquelme,8,Neural Information Processing Systems,1249,"Sparsely-gated Mixture of Experts networks (MoEs) have demonstrated excellent scalability in Natural Language Processing. In Computer Vision, however, almost all performant networks are""dense"", that is, every input is processed by every parameter. We present a Vision MoE (V-MoE), a sparse version of the Vision Transformer, that is scalable and competitive with the largest dense networks. When applied to image recognition, V-MoE matches the performance of state-of-the-art networks, while requiring as little as half of the compute at inference time. Further, we propose an extension to the routing algorithm that can prioritize subsets of each input across the entire batch, leading to adaptive per-image compute. This allows V-MoE to trade-off performance and compute smoothly at test-time. Finally, we demonstrate the potential of V-MoE to scale vision models, and train a 15B parameter model that attains 90.35% on ImageNet." +97d8823ca3c9bd932cec8ad6f3b194168e7cec92,2106.05237,10.1109/CVPR52688.2022.01065,2021,Knowledge distillation: A good teacher is patient and consistent,Lucas Beyer,6,Computer Vision and Pattern Recognition,426,"There is a growing discrepancy in computer vision between large-scale models that achieve state-of-the-art performance and models that are affordable in practical applications. In this paper we address this issue and significantly bridge the gap between these two types of models. Throughout our empirical investigation we do not aim to necessarily propose a new method, but strive to identify a robust and effective recipe for making state-of-the-art large scale models affordable in practice. We demonstrate that, when performed correctly, knowledge distillation can be a powerful tool for reducing the size of large models without compromising their performance. In particular, we uncover that there are certain implicit design choices, which may drastically affect the effectiveness of distillation. Our key contribution is the explicit identification of these design choices, which were not previously articulated in the literature. We back up our findings by a comprehensive empirical study, demonstrate compelling results on a wide range of vision datasets and, in particular, obtain a state-of-the-art ResNet-50 model for ImageNet, which achieves 82.8% top-1 accuracy." +2a805d0e1b067444a554c5169d189fa1f649f411,2106.04560,10.1109/CVPR52688.2022.01179,2021,Scaling Vision Transformers,Xiao-Hua Zhai,4,Computer Vision and Pattern Recognition,1538,"Attention-based neural networks such as the Vision Transformer (ViT) have recently attained state-of-the-art results on many computer vision benchmarks. Scale is a primary ingredient in attaining excellent results, therefore, understanding a model's scaling properties is a key to designing future generations effectively. While the laws for scaling Transformer language models have been studied, it is unknown how Vision Transformers scale. To address this, we scale ViT models and data, both up and down, and characterize the relationships between error rate, data, and compute. Along the way, we refine the architecture and training of ViT, reducing memory consumption and increasing accuracy of the resulting models. As a result, we successfully train a ViT model with two billion parameters, which attains a new state-of-the-art on ImageNet of 90.45% top-1 accuracy. The model also performs well for few-shot transfer, for example, reaching 84.86% top-1 accuracy on ImageNet with only 10 examples per class." +7b9d49b4c18a160512cbe9fc9fa5d8c40cd6b84e,2106.12887,,2021,A Near-Optimal Algorithm for Debiasing Trained Machine Learning Models,Ibrahim M. Alabdulmohsin,2,Neural Information Processing Systems,27,"We present a scalable post-processing algorithm for debiasing trained models, including deep neural networks (DNNs), which we prove to be near-optimal by bounding its excess Bayes risk. We empirically validate its advantages on standard benchmark datasets across both classical algorithms as well as modern DNN architectures and demonstrate that it outperforms previous post-processing methods while performing on par with in-processing. In addition, we show that the proposed algorithm is particularly effective for models trained at scale where post-processing is a natural and practical choice." +c3ddd90b53dc797029c4576af01b9c35bb846a44,2106.03004,,2021,Exploring the Limits of Out-of-Distribution Detection,Stanislav Fort,3,Neural Information Processing Systems,446,"Near out-of-distribution detection (OOD) is a major challenge for deep neural networks. We demonstrate that large-scale pre-trained transformers can significantly improve the state-of-the-art (SOTA) on a range of near OOD tasks across different data modalities. For instance, on CIFAR-100 vs CIFAR-10 OOD detection, we improve the AUROC from 85% (current SOTA) to more than 96% using Vision Transformers pre-trained on ImageNet-21k. On a challenging genomics OOD detection benchmark, we improve the AUROC from 66% to 77% using transformers and unsupervised pre-training. To further improve performance, we explore the few-shot outlier exposure setting where a few examples from outlier classes may be available; we show that pre-trained transformers are particularly well-suited for outlier exposure, and that the AUROC of OOD detection on CIFAR-100 vs CIFAR-10 can be improved to 98.7% with just 1 image per OOD class, and 99.46% with 10 images per OOD class. For multi-modal image-text pre-trained transformers such as CLIP, we explore a new way of using just the names of outlier classes as a sole source of information without any accompanying images, and show that this outperforms previous SOTA on standard vision OOD benchmark tasks." +6ac8e0915df41692bdceef7f18bc1c50271630fc,2105.10305,10.1109/CVPR46437.2021.00160,2021,Correlated Input-Dependent Label Noise in Large-Scale Image Classification,Mark Collier,5,Computer Vision and Pattern Recognition,63,"Large scale image classification datasets often contain noisy labels. We take a principled probabilistic approach to modelling input-dependent, also known as heteroscedastic, label noise in these datasets. We place a multivariate Nor-mal distributed latent variable on the final hidden layer of a neural network classifier. The covariance matrix of this latent variable, models the aleatoric uncertainty due to label noise. We demonstrate that the learned covariance structure captures known sources of label noise between semantically similar and co-occurring classes. Compared to standard neural network training and other baselines, we show significantly improved accuracy on Imagenet ILSVRC 2012 79.3% (+ 2.6%), Imagenet-21k 47.0% (+ 1.1%) and JFT 64.7% (+ 1.6%). We set a new state-of-the-art result on WebVision 1.0 with 76.6% top-1 accuracy. These datasets range from over 1M to over 300M training examples and from 1k classes to more than 21k classes. Our method is simple to use, and we provide an implementation that is a drop-in replacement for the final fully-connected layer in a deep classifier." +68f080e0ac836ea230cb5316fbed273c70422d75,2105.05633,10.1109/ICCV48922.2021.00717,2021,Segmenter: Transformer for Semantic Segmentation,Robin Strudel,4,IEEE International Conference on Computer Vision,2067,"Image segmentation is often ambiguous at the level of individual image patches and requires contextual information to reach label consensus. In this paper we introduce Segmenter, a transformer model for semantic segmentation. In contrast to convolution-based methods, our approach allows to model global context already at the first layer and throughout the network. We build on the recent Vision Transformer (ViT) and extend it to semantic segmentation. To do so, we rely on the output embeddings corresponding to image patches and obtain class labels from these embed-dings with a point-wise linear decoder or a mask trans-former decoder. We leverage models pre-trained for image classification and show that we can fine-tune them on moderate sized datasets available for semantic segmentation. The linear decoder allows to obtain excellent results already, but the performance can be further improved by a mask transformer generating class masks. We conduct an extensive ablation study to show the impact of the different parameters, in particular the performance is better for large models and small patch sizes. Segmenter attains excellent results for semantic segmentation. It outperforms the state of the art on both ADE20K and Pascal Context datasets and is competitive on Cityscapes." +b6382a7351c0c595f91472ac71d3b2d87b3c4844,2103.15691,10.1109/ICCV48922.2021.00676,2021,ViViT: A Video Vision Transformer,Anurag Arnab,6,IEEE International Conference on Computer Vision,3359,"We present pure-transformer based models for video classification, drawing upon the recent success of such models in image classification. Our model extracts spatiotemporal tokens from the input video, which are then encoded by a series of transformer layers. In order to handle the long sequences of tokens encountered in video, we propose several, efficient variants of our model which factorise the spatial- and temporal-dimensions of the input. Although transformer-based models are known to only be effective when large training datasets are available, we show how we can effectively regularise the model during training and leverage pretrained image models to be able to train on comparatively small datasets. We conduct thorough ablation studies, and achieve state-of-the-art results on multiple video classification benchmarks including Kinetics 400 and 600, Epic Kitchens, Something-Something v2 and Moments in Time, outperforming prior methods based on deep 3D convolutional networks." +8e33914d6051dd031a5e096962b9398fc1d16067,2103.13413,10.1109/ICCV48922.2021.01196,2021,Vision Transformers for Dense Prediction,René Ranftl,3,IEEE International Conference on Computer Vision,3214,"We introduce dense prediction transformers, an architecture that leverages vision transformers in place of convolutional networks as a backbone for dense prediction tasks. We assemble tokens from various stages of the vision transformer into image-like representations at various resolutions and progressively combine them into full-resolution predictions using a convolutional decoder. The transformer backbone processes representations at a constant and relatively high resolution and has a global receptive field at every stage. These properties allow the dense prediction transformer to provide finer-grained and more globally coherent predictions when compared to fully-convolutional networks. Our experiments show that this architecture yields substantial improvements on dense prediction tasks, especially when a large amount of training data is available. For monocular depth estimation, we observe an improvement of up to 28% in relative performance when compared to a state-of-the-art fully-convolutional network. When applied to semantic segmentation, dense prediction transformers set a new state of the art on ADE20K with 49.02% mIoU. We further show that the architecture can be fine-tuned on smaller datasets such as NYUv2, KITTI, and Pascal Context where it also sets the new state of the art. Our models are available at https://github.com/intel-isl/DPT." +8c7ee808e2e5ce5bc28f491f3779352aa94be206,2103.03417,10.1145/3461702.3462557,2021,Measuring Model Biases in the Absence of Ground Truth,Osman Aka,5,"AAAI/ACM Conference on AI, Ethics, and Society",43,"The measurement of bias in machine learning often focuses on model performance across identity subgroups (such as man and woman) with respect to groundtruth labels. However, these methods do not directly measure the associations that a model may have learned, for example between labels and identity subgroups. Further, measuring a model's bias requires a fully annotated evaluation dataset which may not be easily available in practice. We present an elegant mathematical solution that tackles both issues simultaneously, using image classification as a working example. By treating a classification model's predictions for a given image as a set of labels analogous to a ""bag of words"", we rank the biases that a model has learned with respect to different identity labels. We use man, woman as a concrete example of an identity label set (although this set need not be binary), and present rankings for the labels that are most biased towards one identity or the other. We demonstrate how the statistical properties of different association metrics can lead to different rankings of the most ""gender biased"" labels, and conclude that normalized pointwise mutual information (nPMI) is most useful in practice. Finally, we announce an open-sourced nPMI visualization tool using TensorBoard." +6f870f7f02a8c59c3e23f407f3ef00dd1dcf8fc4,2103.00020,,2021,Learning Transferable Visual Models From Natural Language Supervision,A. Radford,12,International Conference on Machine Learning,58740,"State-of-the-art computer vision systems are trained to predict a fixed set of predetermined object categories. This restricted form of supervision limits their generality and usability since additional labeled data is needed to specify any other visual concept. Learning directly from raw text about images is a promising alternative which leverages a much broader source of supervision. We demonstrate that the simple pre-training task of predicting which caption goes with which image is an efficient and scalable way to learn SOTA image representations from scratch on a dataset of 400 million (image, text) pairs collected from the internet. After pre-training, natural language is used to reference learned visual concepts (or describe new ones) enabling zero-shot transfer of the model to downstream tasks. We study the performance of this approach by benchmarking on over 30 different existing computer vision datasets, spanning tasks such as OCR, action recognition in videos, geo-localization, and many types of fine-grained object classification. The model transfers non-trivially to most tasks and is often competitive with a fully supervised baseline without the need for any dataset specific training. For instance, we match the accuracy of the original ResNet-50 on ImageNet zero-shot without needing to use any of the 1.28 million training examples it was trained on. We release our code and pre-trained model weights at https://github.com/OpenAI/CLIP." +141a5033d9994242b18bb3b217e79582f1ee9306,2102.05918,,2021,Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision,Chao Jia,10,International Conference on Machine Learning,6175,"Pre-trained representations are becoming crucial for many NLP and perception tasks. While representation learning in NLP has transitioned to training on raw text without human annotations, visual and vision-language representations still rely heavily on curated training datasets that are expensive or require expert knowledge. For vision applications, representations are mostly learned using datasets with explicit class labels such as ImageNet or OpenImages. For vision-language, popular datasets like Conceptual Captions, MSCOCO, or CLIP all involve a non-trivial data collection (and cleaning) process. This costly curation process limits the size of datasets and hence hinders the scaling of trained models. In this paper, we leverage a noisy dataset of over one billion image alt-text pairs, obtained without expensive filtering or post-processing steps in the Conceptual Captions dataset. A simple dual-encoder architecture learns to align visual and language representations of the image and text pairs using a contrastive loss. We show that the scale of our corpus can make up for its noise and leads to state-of-the-art representations even with such a simple learning scheme. Our visual representation achieves strong performance when transferred to classification tasks such as ImageNet and VTAB. The aligned visual and language representations enables zero-shot image classification and also set new state-of-the-art results on Flickr30K and MSCOCO image-text retrieval benchmarks, even when compared with more sophisticated cross-attention models. The representations also enable cross-modality search with complex text and text + image queries." +fdacf2a732f55befdc410ea927091cad3b791f13,2101.03961,,2021,Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity,W. Fedus,3,Journal of machine learning research,5199,"In deep learning, models typically reuse the same parameters for all inputs. Mixture of Experts (MoE) defies this and instead selects different parameters for each incoming example. The result is a sparsely-activated model -- with outrageous numbers of parameters -- but a constant computational cost. However, despite several notable successes of MoE, widespread adoption has been hindered by complexity, communication costs and training instability -- we address these with the Switch Transformer. We simplify the MoE routing algorithm and design intuitive improved models with reduced communication and computational costs. Our proposed training techniques help wrangle the instabilities and we show large sparse models may be trained, for the first time, with lower precision (bfloat16) formats. We design models based off T5-Base and T5-Large to obtain up to 7x increases in pre-training speed with the same computational resources. These improvements extend into multilingual settings where we measure gains over the mT5-Base version across all 101 languages. Finally, we advance the current scale of language models by pre-training up to trillion parameter models on the""Colossal Clean Crawled Corpus""and achieve a 4x speedup over the T5-XXL model." +c8fa8301190a6e1bc2f3696bec4114da71e70f06,2011.11311,,2020,Uncovering the Bias in Facial Expressions,Jessica Deuschel,3,arXiv.org,15,"Over the past decades the machine and deep learning community has celebrated great achievements in challenging tasks such as image classification. The deep architecture of artificial neural networks together with the plenitude of available data makes it possible to describe highly complex relations. Yet, it is still impossible to fully capture what the deep learning model has learned and to verify that it operates fairly and without creating bias, especially in critical tasks, for instance those arising in the medical field. One example for such a task is the detection of distinct facial expressions, called Action Units, in facial images. Considering this specific task, our research aims to provide transparency regarding bias, specifically in relation to gender and skin color. We train a neural network for Action Unit classification and analyze its performance quantitatively based on its accuracy and qualitatively based on heatmaps. A structured review of our results indicates that we are able to detect bias. Even though we cannot conclude from our results that lower classification performance emerged solely from gender and skin color bias, these biases must be addressed, which is why we end by giving suggestions on how the detected bias can be avoided." +268d347e8a55b5eb82fb5e7d2f800e33c75ab18a,2010.11929,,2020,An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale,A. Dosovitskiy,12,International Conference on Learning Representations,72694,"While the Transformer architecture has become the de-facto standard for natural language processing tasks, its applications to computer vision remain limited. In vision, attention is either applied in conjunction with convolutional networks, or used to replace certain components of convolutional networks while keeping their overall structure in place. We show that this reliance on CNNs is not necessary and a pure transformer applied directly to sequences of image patches can perform very well on image classification tasks. When pre-trained on large amounts of data and transferred to multiple mid-sized or small image recognition benchmarks (ImageNet, CIFAR-100, VTAB, etc.), Vision Transformer (ViT) attains excellent results compared to state-of-the-art convolutional networks while requiring substantially fewer computational resources to train." +2999921763d317c253332289500a2e24c44f28e1,2010.06498,,2020,Cross-Domain Few-Shot Learning by Representation Fusion,Thomas Adler,8,arXiv.org,50,"In order to quickly adapt to new data, few-shot learning aims at learning from few examples, often by using already acquired knowledge. The new data often differs from the previously seen data due to a domain shift, that is, a change of the input-target distribution. While several methods perform well on small domain shifts like new target classes with similar inputs, larger domain shifts are still challenging. Large domain shifts may result in high-level concepts that are not shared between the original and the new domain. However, low-level concepts like edges in images might still be shared and useful. For cross-domain few-shot learning, we suggest representation fusion to unify different abstraction levels of a deep neural network into one representation. We propose Cross-domain Hebbian Ensemble Few-shot learning (CHEF), which achieves representation fusion by an ensemble of Hebbian learners acting on different layers of a deep neural network that was trained on the original domain. On the few-shot datasets miniImagenet and tieredImagenet, where the domain shift is small, CHEF is competitive with state-of-the-art methods. On cross-domain few-shot benchmark challenges with larger domain shifts, CHEF establishes novel state-of-the-art results in all categories. We further apply CHEF on a real-world cross-domain application in drug discovery. We consider a domain shift from bioactive molecules to environmental chemicals and drugs with twelve associated toxicity prediction tasks. On these tasks, that are highly relevant for computational drug discovery, CHEF significantly outperforms all its competitors. Github: this https URL" +948a09558dc0022ce3833de4e7a9b7610ee4b791,2008.07246,10.5194/isprs-annals-V-2-2020-357-2020,2020,Self-Supervised Learning for Monocular Depth Estimation from Aerial Imagery,M. Hermann,4,"ISPRS Annals of the Photogrammetry, Remote Sensing and Spatial Information Sciences",16,"Abstract. Supervised learning based methods for monocular depth estimation usually require large amounts of extensively annotated training data. In the case of aerial imagery, this ground truth is particularly difficult to acquire. Therefore, in this paper, we present a method for self-supervised learning for monocular depth estimation from aerial imagery that does not require annotated training data. For this, we only use an image sequence from a single moving camera and learn to simultaneously estimate depth and pose information. By sharing the weights between pose and depth estimation, we achieve a relatively small model, which favors real-time application. We evaluate our approach on three diverse datasets and compare the results to conventional methods that estimate depth maps based on multi-view geometry. We achieve an accuracy δ1:25 of up to 93.5 %. In addition, we have paid particular attention to the generalization of a trained model to unknown data and the self-improving capabilities of our approach. We conclude that, even though the results of monocular depth estimation are inferior to those achieved by conventional methods, they are well suited to provide a good initialization for methods that rely on image matching or to provide estimates in regions where image matching fails, e.g. occluded or texture-less regions." +0593d3da080f886fa020541a1e1c675f4fdd37c6,2007.08558,10.1109/CVPR46437.2021.01619,2020,On Robustness and Transferability of Convolutional Neural Networks,J. Djolonga,14,Computer Vision and Pattern Recognition,180,"Modern deep convolutional networks (CNNs) are often criticized for not generalizing under distributional shifts. However, several recent breakthroughs in transfer learning suggest that these networks can cope with severe distribution shifts and successfully adapt to new tasks from a few training examples. In this work we study the interplay between out-of-distribution and transfer performance of modern image classification CNNs for the first time and investigate the impact of the pre-training data size, the model scale, and the data preprocessing pipeline. We find that increasing both the training set and model sizes significantly improve the distributional shift robustness. Furthermore, we show that, perhaps surprisingly, simple changes in the preprocessing such as modifying the image resolution can significantly mitigate robustness issues in some cases. Finally, we outline the shortcomings of existing robustness evaluation datasets and introduce a synthetic dataset SI-SCORE we use for a systematic analysis across factors of variation common in visual data such as object size and position." +e609165fac8e4923423f754fb82292a614395c92,2007.00644,,2020,Measuring Robustness to Natural Distribution Shifts in Image Classification,Rohan Taori,6,Neural Information Processing Systems,741,"We study how robust current ImageNet models are to distribution shifts arising from natural variations in datasets. Most research on robustness focuses on synthetic image perturbations (noise, simulated weather artifacts, adversarial examples, etc.), which leaves open how robustness on synthetic distribution shift relates to distribution shift arising in real data. Informed by an evaluation of 204 ImageNet models in 213 different test conditions, we find that there is often little to no transfer of robustness from current synthetic to natural distribution shift. Moreover, most current techniques provide no robustness to the natural distribution shifts in our testbed. The main exception is training on larger and more diverse datasets, which in multiple cases increases robustness, but is still far from closing the performance gaps. Our results indicate that distribution shifts arising in real data are currently an open research problem. We provide our testbed and data as a resource for future work at this https URL ." +022622e024890d6e044ac50e2da6b44c59bdf418,2006.16241,10.1109/ICCV48922.2021.00823,2020,The Many Faces of Robustness: A Critical Analysis of Out-of-Distribution Generalization,Dan Hendrycks,13,IEEE International Conference on Computer Vision,2597,"We introduce four new real-world distribution shift datasets consisting of changes in image style, image blurriness, geographic location, camera operation, and more. With our new datasets, we take stock of previously proposed methods for improving out-of-distribution robustness and put them to the test. We find that using larger models and artificial data augmentations can improve robustness on real-world distribution shifts, contrary to claims in prior work. We find improvements in artificial robustness benchmarks can transfer to real-world distribution shifts, contrary to claims in prior work. Motivated by our observation that data augmentations can help with real-world distribution shifts, we also introduce a new data augmentation method which advances the state-of-the-art and outperforms models pre-trained with 1000× more labeled data. Overall we find that some methods consistently help with distribution shifts in texture and local image statistics, but these methods do not help with some other distribution shifts like geographic changes. Our results show that future research must study multiple distribution shifts simultaneously, as we demonstrate that no evaluated method consistently improves robustness." +ab5f0004c5f3317689e8457e1c8d8390ccbee522,,10.1145/3360307,2020,A domain-specific supercomputer for training deep neural networks,N. Jouppi,8,Communications of the ACM,310,Google's TPU supercomputers train deep neural networks 50x faster than general-purpose supercomputers running a high-performance computing benchmark. +f5c8464032a936451b222be1984cabf42d6adfa8,2006.07159,,2020,Are we done with ImageNet?,Lucas Beyer,5,arXiv.org,503,"Yes, and no. We ask whether recent progress on the ImageNet classification benchmark continues to represent meaningful generalization, or whether the community has started to overfit to the idiosyncrasies of its labeling procedure. We therefore develop a significantly more robust procedure for collecting human annotations of the ImageNet validation set. Using these new labels, we reassess the accuracy of recently proposed ImageNet classifiers, and find their gains to be substantially smaller than those reported on the original labels. Furthermore, we find the original ImageNet labels to no longer be the best predictors of this independently-collected set, indicating that their usefulness in evaluating vision models may be nearing an end. Nevertheless, we find our annotation procedure to have largely remedied the errors in the original labels, reinforcing ImageNet as a powerful benchmark for future research in visual recognition." +76a9f336481b39515d6cea2920696f11fb686451,2005.00928,10.18653/v1/2020.acl-main.385,2020,Quantifying Attention Flow in Transformers,Samira Abnar,2,Annual Meeting of the Association for Computational Linguistics,1527,"In the Transformer model, “self-attention” combines information from attended embeddings into the representation of the focal embedding in the next layer. Thus, across layers of the Transformer, information originating from different tokens gets increasingly mixed. This makes attention weights unreliable as explanations probes. In this paper, we consider the problem of quantifying this flow of information through self-attention. We propose two methods for approximating the attention to input tokens given attention weights, attention rollout and attention flow, as post hoc methods when we use attention weights as the relative relevance of the input tokens. We show that these methods give complementary views on the flow of information, and compared to raw attention, both yield higher correlations with importance scores of input tokens obtained using an ablation method and input gradients." +b145ea2049648535d6081407ebd315b072248183,2002.06715,,2020,BatchEnsemble: An Alternative Approach to Efficient Ensemble and Lifelong Learning,Yeming Wen,3,International Conference on Learning Representations,610,"Ensembles, where multiple neural networks are trained individually and their predictions are averaged, have been shown to be widely successful for improving both the accuracy and predictive uncertainty of single neural networks. However, an ensemble's cost for both training and testing increases linearly with the number of networks, which quickly becomes untenable. In this paper, we propose BatchEnsemble, an ensemble method whose computational and memory costs are significantly lower than typical ensembles. BatchEnsemble achieves this by defining each weight matrix to be the Hadamard product of a shared weight among all ensemble members and a rank-one matrix per member. Unlike ensembles, BatchEnsemble is not only parallelizable across devices, where one device trains one member, but also parallelizable within a device, where multiple ensemble members are updated simultaneously for a given mini-batch. Across CIFAR-10, CIFAR-100, WMT14 EN-DE/EN-FR translation, and out-of-distribution tasks, BatchEnsemble yields competitive accuracy and uncertainties as typical ensembles; the speedup at test time is 3X and memory reduction is 3X at an ensemble of size 4. We also apply BatchEnsemble to lifelong learning, where on Split-CIFAR-100, BatchEnsemble yields comparable performance to progressive neural networks while having a much lower computational and memory costs. We further show that BatchEnsemble can easily scale up to lifelong learning on Split-ImageNet which involves 100 sequential learning tasks." +bc51622358d8eea83248ef29402fe10640d07ba6,1912.11370,10.1007/978-3-030-58558-7_29,2019,Big Transfer (BiT): General Visual Representation Learning,Alexander Kolesnikov,7,European Conference on Computer Vision,1425,"Transfer of pre-trained representations improves sample efficiency and simplifies hyperparameter tuning when training deep neural networks for vision. We revisit the paradigm of pre-training on large supervised datasets and fine-tuning the model on a target task. We scale up pre-training, and propose a simple recipe that we call Big Transfer (BiT). By combining a few carefully selected components, and transferring using a simple heuristic, we achieve strong performance on over 20 datasets. BiT performs well across a surprisingly wide range of data regimes -- from 1 example per class to 1M total examples. BiT achieves 87.5% top-1 accuracy on ILSVRC-2012, 99.4% on CIFAR-10, and 76.3% on the 19 task Visual Task Adaptation Benchmark (VTAB). On small datasets, BiT attains 76.8% on ILSVRC-2012 with 10 examples per class, and 97.0% on CIFAR-10 with 10 examples per class. We conduct detailed analysis of the main components that lead to high transfer performance." +8406903fd2f0eb25349bf071ccfaae3947e2a9cd,1912.04838,10.1109/CVPR42600.2020.00252,2019,Scalability in Perception for Autonomous Driving: Waymo Open Dataset,Pei Sun,25,Computer Vision and Pattern Recognition,4603,"The research community has increasing interest in autonomous driving research, despite the resource intensity of obtaining representative real world data. Existing self-driving datasets are limited in the scale and variation of the environments they capture, even though generalization within and between operating regions is crucial to the over-all viability of the technology. In an effort to help align the research community’s contributions with real-world self-driving problems, we introduce a new large scale, high quality, diverse dataset. Our new dataset consists of 1150 scenes that each span 20 seconds, consisting of well synchronized and calibrated high quality LiDAR and camera data captured across a range of urban and suburban geographies. It is 15x more diverse than the largest camera+LiDAR dataset available based on our proposed diversity metric. We exhaustively annotated this data with 2D (camera image) and 3D (LiDAR) bounding boxes, with consistent identifiers across frames. Finally, we provide strong baselines for 2D as well as 3D detection and tracking tasks. We further study the effects of dataset size and generalization across geographies on 3D detection methods. Find data, code and more up-to-date information at http://www.waymo.com/open." +17d50efa0d5ad863f5939eb586bb0a5436b0adf7,1911.11834,10.1109/cvpr42600.2020.00894,2019,Towards Fairness in Visual Recognition: Effective Strategies for Bias Mitigation,Zeyu Wang,7,Computer Vision and Pattern Recognition,435,"Computer vision models learn to perform a task by capturing relevant statistics from training data. It has been shown that models learn spurious age, gender, and race correlations when trained for seemingly unrelated tasks like activity recognition or image captioning. Various mitigation techniques have been presented to prevent models from utilizing or learning such biases. However, there has been little systematic comparison between these techniques. We design a simple but surprisingly effective visual recognition benchmark for studying bias mitigation. Using this benchmark, we provide a thorough analysis of a wide range of techniques. We highlight the shortcomings of popular adversarial training approaches for bias mitigation, propose a simple but similarly effective alternative to the inference-time Reducing Bias Amplification method of Zhao et al., and design a domain-independent training technique that outperforms all other methods. Finally, we validate our findings on the attribute classification task in the CelebA dataset, where attribute presence is known to be correlated with the gender of people in the image, and demonstrate that the proposed technique is effective at mitigating real-world gender bias." +3bc53c49ae68adacf2d5be2fa795bcb879e2717a,1911.09483,,2019,MUSE: Parallel Multi-Scale Attention for Sequence to Sequence Learning,Guangxiang Zhao,5,arXiv.org,62,"In sequence to sequence learning, the self-attention mechanism proves to be highly effective, and achieves significant improvements in many tasks. However, the self-attention mechanism is not without its own flaws. Although self-attention can model extremely long dependencies, the attention in deep layers tends to overconcentrate on a single token, leading to insufficient use of local information and difficultly in representing long sequences. In this work, we explore parallel multi-scale representation learning on sequence data, striving to capture both long-range and short-range language structures. To this end, we propose the Parallel MUlti-Scale attEntion (MUSE) and MUSE-simple. MUSE-simple contains the basic idea of parallel multi-scale sequence representation learning, and it encodes the sequence in parallel, in terms of different scales with the help from self-attention, and pointwise transformation. MUSE builds on MUSE-simple and explores combining convolution and self-attention for learning sequence representations from more different scales. We focus on machine translation and the proposed approach achieves substantial performance improvements over Transformer, especially on long sequences. More importantly, we find that although conceptually simple, its success in practice requires intricate considerations, and the multi-scale attention must build on unified semantic space. Under common setting, the proposed model achieves substantial performance and outperforms all previous models on three main machine translation tasks. In addition, MUSE has potential for accelerating inference due to its parallelism. Code will be available at this https URL" +bb713d56a39a040b35e4f9e036fb4422f543e614,1911.03584,,2019,On the Relationship between Self-Attention and Convolutional Layers,Jean-Baptiste Cordonnier,3,International Conference on Learning Representations,666,"Recent trends of incorporating attention mechanisms in vision have led researchers to reconsider the supremacy of convolutional layers as a primary building block. Beyond helping CNNs to handle long-range dependencies, Ramachandran et al. (2019) showed that attention can completely replace convolution and achieve state-of-the-art performance on vision tasks. This raises the question: do learned attention layers operate similarly to convolutional layers? This work provides evidence that attention layers can perform convolution and, indeed, they often learn to do so in practice. Specifically, we prove that a multi-head self-attention layer with sufficient number of heads is at least as expressive as any convolutional layer. Our numerical experiments then show that self-attention layers attend to pixel-grid patterns similarly to CNN layers, corroborating our analysis. Our code is publicly available." +6c4b76232bb72897685d19b3d264c6ee3005bc2b,1910.10683,,2019,Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer,Colin A. Raffel,9,Journal of machine learning research,27969,"Transfer learning, where a model is first pre-trained on a data-rich task before being fine-tuned on a downstream task, has emerged as a powerful technique in natural language processing (NLP). The effectiveness of transfer learning has given rise to a diversity of approaches, methodology, and practice. In this paper, we explore the landscape of transfer learning techniques for NLP by introducing a unified framework that converts every language problem into a text-to-text format. Our systematic study compares pre-training objectives, architectures, unlabeled datasets, transfer approaches, and other factors on dozens of language understanding tasks. By combining the insights from our exploration with scale and our new ""Colossal Clean Crawled Corpus"", we achieve state-of-the-art results on many benchmarks covering summarization, question answering, text classification, and more. To facilitate future work on transfer learning for NLP, we release our dataset, pre-trained models, and code." +10eda4521c032adabaa8e70d6569e17370b29dcd,1910.07467,10.5167/UZH-177483,2019,Root Mean Square Layer Normalization,Biao Zhang,2,Neural Information Processing Systems,2159,"Layer normalization (LayerNorm) has been successfully applied to various deep neural networks to help stabilize training and boost model convergence because of its capability in handling re-centering and re-scaling of both inputs and weight matrix. However, the computational overhead introduced by LayerNorm makes these improvements expensive and significantly slows the underlying network, e.g. RNN in particular. In this paper, we hypothesize that re-centering invariance in LayerNorm is dispensable and propose root mean square layer normalization, or RMSNorm. RMSNorm regularizes the summed inputs to a neuron in one layer according to root mean square (RMS), giving the model re-scaling invariance property and implicit learning rate adaptation ability. RMSNorm is computationally simpler and thus more efficient than LayerNorm. We also present partial RMSNorm, or pRMSNorm where the RMS is estimated from p% of the summed inputs without breaking the above properties. Extensive experiments on several tasks using diverse network architectures show that RMSNorm achieves comparable performance against LayerNorm but reduces the running time by 7%~64% on different models. Source code is available at https://github.com/bzhangGo/rmsnorm." +85b9e68eb27069e87181050035f40b79438dd220,1910.04867,,2019,A Large-scale Study of Representation Learning with the Visual Task Adaptation Benchmark,Xiao-Hua Zhai,17,,586,"Representation learning promises to unlock deep learning for the long tail of vision tasks without expensive labelled datasets. Yet, the absence of a unified evaluation for general visual representations hinders progress. Popular protocols are often too constrained (linear classification), limited in diversity (ImageNet, CIFAR, Pascal-VOC), or only weakly related to representation quality (ELBO, reconstruction error). We present the Visual Task Adaptation Benchmark (VTAB), which defines good representations as those that adapt to diverse, unseen tasks with few examples. With VTAB, we conduct a large-scale study of many popular publicly-available representation learning algorithms. We carefully control confounders such as architecture and tuning budget. We address questions like: How effective are ImageNet representations beyond standard natural datasets? How do representations trained via generative and discriminative models compare? To what extent can self-supervision replace labels? And, how close are we to general visual representations?" +45557cc70cd6989ab6b03e5aeb787e34299099f7,1907.07174,10.1109/CVPR46437.2021.01501,2019,Natural Adversarial Examples,Dan Hendrycks,5,Computer Vision and Pattern Recognition,2103,"We introduce two challenging datasets that reliably cause machine learning model performance to substantially degrade. The datasets are collected with a simple adversarial filtration technique to create datasets with limited spurious cues. Our datasets’ real-world, unmodified examples transfer to various unseen models reliably, demonstrating that computer vision models have shared weaknesses. The first dataset is called IMAGENET-A and is like the ImageNet test set, but it is far more challenging for existing models. We also curate an adversarial out-of-distribution detection dataset called IMAGENET-O, which is the first out-of-distribution detection dataset created for ImageNet models. On IMAGENET-A a DenseNet-121 obtains around 2% accuracy, an accuracy drop of approximately 90%, and its out-of-distribution detection performance on IMAGENET-O is near random chance levels. We find that existing data augmentation techniques hardly boost performance, and using other public training datasets provides improvements that are limited. However, we find that improvements to computer vision architectures provide a promising path towards robust models." +9ff54196b2505ed622ad4d0025c4ac96d012d0a4,,,2019,Metric Learning for Patch Classification in Digital Pathology,Eu Wern Teh,2,,12, +49b64383fe36268410c430352637ed23b16820c5,1903.12261,,2019,Benchmarking Neural Network Robustness to Common Corruptions and Perturbations,Dan Hendrycks,2,International Conference on Learning Representations,4937,"In this paper we establish rigorous benchmarks for image classifier robustness. Our first benchmark, ImageNet-C, standardizes and expands the corruption robustness topic, while showing which classifiers are preferable in safety-critical applications. Then we propose a new dataset called ImageNet-P which enables researchers to benchmark a classifier's robustness to common perturbations. Unlike recent robustness research, this benchmark evaluates performance on common corruptions and perturbations not worst-case adversarial perturbations. We find that there are negligible changes in relative corruption robustness from AlexNet classifiers to ResNet classifiers. Afterward we discover ways to enhance corruption and perturbation robustness. We even find that a bypassed adversarial defense provides substantial common perturbation robustness. Together our benchmarks may aid future work toward networks that robustly generalize." +4e0bb8c1c683b43357c5d5216f6b74ff2cb32434,1902.10811,,2019,Do ImageNet Classifiers Generalize to ImageNet?,B. Recht,4,International Conference on Machine Learning,2450,"We build new test sets for the CIFAR-10 and ImageNet datasets. Both benchmarks have been the focus of intense research for almost a decade, raising the danger of overfitting to excessively re-used test sets. By closely following the original dataset creation processes, we test to what extent current classification models generalize to new data. We evaluate a broad range of models and find accuracy drops of 3% - 15% on CIFAR-10 and 11% - 14% on ImageNet. However, accuracy gains on the original test sets translate to larger gains on the new test sets. Our results suggest that the accuracy drops are not caused by adaptivity, but by the models' inability to generalize to slightly ""harder"" images than those found in the original test sets." +7365f887c938ca21a6adbef08b5a520ebbd4638f,1810.03993,10.1145/3287560.3287596,2018,Model Cards for Model Reporting,Margaret Mitchell,9,FAT,3452,"Trained machine learning models are increasingly used to perform high-impact tasks in areas such as law enforcement, medicine, education, and employment. In order to clarify the intended use cases of machine learning models and minimize their usage in contexts for which they are not well suited, we recommend that released models be accompanied by documentation detailing their performance characteristics. In this paper, we propose a framework that we call model cards, to encourage such transparent model reporting. Model cards are short documents accompanying trained machine learning models that provide benchmarked evaluation in a variety of conditions, such as across different cultural, demographic, or phenotypic groups (e.g., race, geographic location, sex, Fitzpatrick skin type [15]) and intersectional groups (e.g., age and race, or sex and Fitzpatrick skin type) that are relevant to the intended application domains. Model cards also disclose the context in which models are intended to be used, details of the performance evaluation procedures, and other relevant information. While we focus primarily on human-centered machine learning models in the application fields of computer vision and natural language processing, this framework can be used to document any trained machine learning model. To solidify the concept, we provide cards for two supervised models: One trained to detect smiling faces in images, and one trained to detect toxic comments in text. We propose model cards as a step towards the responsible democratization of machine learning and related artificial intelligence technology, increasing transparency into how well artificial intelligence technology works. We hope this work encourages those releasing trained machine learning models to accompany model releases with similar detailed evaluation numbers and other relevant documentation." +0f50b7483f1b200ebf88c4dd7698de986399a0f3,1811.12231,,2018,ImageNet-trained CNNs are biased towards texture; increasing shape bias improves accuracy and robustness,Robert Geirhos,6,International Conference on Learning Representations,3377,"Convolutional Neural Networks (CNNs) are commonly thought to recognise objects by learning increasingly complex representations of object shapes. Some recent studies suggest a more important role of image textures. We here put these conflicting hypotheses to a quantitative test by evaluating CNNs and human observers on images with a texture-shape cue conflict. We show that ImageNet-trained CNNs are strongly biased towards recognising textures rather than shapes, which is in stark contrast to human behavioural evidence and reveals fundamentally different classification strategies. We then demonstrate that the same standard architecture (ResNet-50) that learns a texture-based representation on ImageNet is able to learn a shape-based representation instead when trained on ""Stylized-ImageNet"", a stylized version of ImageNet. This provides a much better fit for human behavioural performance in our well-controlled psychophysical lab setting (nine experiments totalling 48,560 psychophysical trials across 97 observers) and comes with a number of unexpected emergent benefits such as improved object detection performance and previously unseen robustness towards a wide range of image distortions, highlighting advantages of a shape-based representation." +b5246fa284f86b544a7c31f050b3bd0defd053fd,1808.06226,10.18653/v1/D18-2012,2018,SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processing,Taku Kudo,2,Conference on Empirical Methods in Natural Language Processing,4390,"This paper describes SentencePiece, a language-independent subword tokenizer and detokenizer designed for Neural-based text processing, including Neural Machine Translation. It provides open-source C++ and Python implementations for subword units. While existing subword segmentation tools assume that the input is pre-tokenized into word sequences, SentencePiece can train subword models directly from raw sentences, which allows us to make a purely end-to-end and language independent system. We perform a validation experiment of NMT on English-Japanese machine translation, and find that it is possible to achieve comparable accuracy to direct subword training from raw sentences. We also compare the performance of subword training and segmentation with various configurations. SentencePiece is available under the Apache 2 license at https://github.com/google/sentencepiece." +aaab0bd4d79d4f19109bab0fbcdb05070fb0edd1,1807.10221,10.1007/978-3-030-01228-1_26,2018,Unified Perceptual Parsing for Scene Understanding,Te-Te Xiao,5,European Conference on Computer Vision,2713,"Humans recognize the visual world at multiple levels: we effortlessly categorize scenes and detect objects inside, while also identifying the textures and surfaces of the objects along with their different compositional parts. In this paper, we study a new task called Unified Perceptual Parsing, which requires the machine vision systems to recognize as many visual concepts as possible from a given image. A multi-task framework called UPerNet and a training strategy are developed to learn from heterogeneous image annotations. We benchmark our framework on Unified Perceptual Parsing and show that it is able to effectively segment a wide range of concepts from images. The trained networks are further applied to discover visual knowledge in natural scenes (Models are available at https://github.com/CSAILVision/unifiedparsing)." +f986968735459e789890f24b6b277b0920a9725d,,10.1109/TPAMI.2017.2723009,2018,Places: A 10 Million Image Database for Scene Recognition,Bolei Zhou,5,IEEE Transactions on Pattern Analysis and Machine Intelligence,5023,"The rise of multi-million-item dataset initiatives has enabled data-hungry machine learning algorithms to reach near-human semantic classification performance at tasks such as visual object and scene recognition. Here we describe the Places Database, a repository of 10 million scene photographs, labeled with scene semantic categories, comprising a large and diverse list of the types of environments encountered in the world. Using the state-of-the-art Convolutional Neural Networks (CNNs), we provide scene classification CNNs (Places-CNNs) as baselines, that significantly outperform the previous approaches. Visualization of the CNNs trained on Places shows that object detectors emerge as an intermediate representation of scene classification. With its high-coverage and high-diversity of exemplars, the Places Database along with the Places-CNNs offer a novel resource to guide future progress on scene recognition problems." +89c3355f5bc7130ae4ed090c8accc52dd885d558,1806.06193,10.1109/CVPR.2018.00432,2018,Large Scale Fine-Grained Categorization and Domain-Specific Transfer Learning,Yin Cui,5,2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition,543,"Transferring the knowledge learned from large scale datasets (e.g., ImageNet) via fine-tuning offers an effective solution for domain-specific fine-grained visual categorization (FGVC) tasks (e.g., recognizing bird species or car make & model). In such scenarios, data annotation often calls for specialized domain knowledge and thus is difficult to scale. In this work, we first tackle a problem in large scale FGVC. Our method won first place in iNaturalist 2017 large scale species classification challenge. Central to the success of our approach is a training scheme that uses higher image resolution and deals with the long-tailed distribution of training data. Next, we study transfer learning via fine-tuning from large scale datasets to small scale, domain-specific FGVC datasets. We propose a measure to estimate domain similarity via Earth Mover's Distance and demonstrate that transfer learning benefits from pre-training on a source domain that is similar to the target domain by this measure. Our proposed transfer learning outperforms ImageNet pre-training and obtains state-of-the-art results on multiple commonly used FGVC datasets." +0f885fd46064d271d4404cf9bb3d758e1a6f8d55,1805.00932,10.1007/978-3-030-01216-8_12,2018,Exploring the Limits of Weakly Supervised Pretraining,D. Mahajan,8,European Conference on Computer Vision,1496,"State-of-the-art visual perception models for a wide range of tasks rely on supervised pretraining. ImageNet classification is the de facto pretraining task for these models. Yet, ImageNet is now nearly ten years old and is by modern standards ""small"". Even so, relatively little is known about the behavior of pretraining with datasets that are multiple orders of magnitude larger. The reasons are obvious: such datasets are difficult to collect and annotate. In this paper, we present a unique study of transfer learning with large convolutional networks trained to predict hashtags on billions of social media images. Our experiments demonstrate that training for large-scale hashtag prediction leads to excellent results. We show improvements on several image classification and object detection tasks, and report the highest ImageNet-1k single-crop, top-1 accuracy to date: 85.4% (97.6% top-5). We also perform extensive experiments that provide novel empirical data on the relationship between large-scale pretraining and transfer learning performance." +b0c5dc3fa19a2bc97606ccb6f55226b913984395,1803.09797,10.1007/978-3-030-01219-9_47,2018,Women also Snowboard: Overcoming Bias in Captioning Models,Lisa Anne Hendricks,5,European Conference on Computer Vision,534,"Most machine learning methods are known to capture and exploit biases of the training data. While some biases are beneficial for learning, others are harmful. Specifically, image captioning models tend to exaggerate biases present in training data (e.g., if a word is present in 60% of training sentences, it might be predicted in 70% of sentences at test time). This can lead to incorrect captions in domains where unbiased captions are desired, or required, due to over-reliance on the learned prior and image context. In this work we investigate generation of gender-specific caption words (e.g. man, woman) based on the person's appearance or the image context. We introduce a new Equalizer model that ensures equal gender probability when gender evidence is occluded in a scene and confident predictions when gender evidence is present. The resulting model is forced to look at a person rather than use contextual cues to make a gender-specific predictions. The losses that comprise our model, the Appearance Confusion Loss and the Confident Loss, are general, and can be added to any description model in order to mitigate impacts of unwanted bias in a description dataset. Our proposed model has lower error than prior work when describing images with people and mentioning their gender and more closely matches the ground truth ratio of sentences including women to sentences including men. We also show that unlike other approaches, our model is indeed more often looking at people when predicting their gender." +18858cc936947fc96b5c06bbe3c6c2faa5614540,,,2018,Gender Shades: Intersectional Accuracy Disparities in Commercial Gender Classification,J. Buolamwini,2,FAT,5780, +c468bbde6a22d961829e1970e6ad5795e05418d1,1801.03924,10.1109/CVPR.2018.00068,2018,The Unreasonable Effectiveness of Deep Features as a Perceptual Metric,Richard Zhang,5,2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition,21027,"While it is nearly effortless for humans to quickly assess the perceptual similarity between two images, the underlying processes are thought to be quite complex. Despite this, the most widely used perceptual metrics today, such as PSNR and SSIM, are simple, shallow functions, and fail to account for many nuances of human perception. Recently, the deep learning community has found that features of the VGG network trained on ImageNet classification has been remarkably useful as a training loss for image synthesis. But how perceptual are these so-called ""perceptual losses""? What elements are critical for their success? To answer these questions, we introduce a new dataset of human perceptual similarity judgments. We systematically evaluate deep features across different architectures and tasks and compare them with classic metrics. We find that deep features outperform all previous metrics by large margins on our dataset. More surprisingly, this result is not restricted to ImageNet-trained VGG features, but holds across different deep architectures and levels of supervision (supervised, self-supervised, or even unsupervised). Our results suggest that perceptual similarity is an emergent property shared across deep visual representations." +07174c2f209f15cacf9ad3422b48652df286be69,1801.03150,10.1109/TPAMI.2019.2901464,2018,Moments in Time Dataset: One Million Videos for Event Understanding,Mathew Monfort,11,IEEE Transactions on Pattern Analysis and Machine Intelligence,643,"We present the Moments in Time Dataset, a large-scale human-annotated collection of one million short videos corresponding to dynamic events unfolding within three seconds. Modeling the spatial-audio-temporal dynamics even for actions occurring in 3 second videos poses many challenges: meaningful events do not include only people, but also objects, animals, and natural phenomena; visual and auditory events can be symmetrical in time (“opening” is “closing” in reverse), and either transient or sustained. We describe the annotation process of our dataset (each video is tagged with one action or activity label among 339 different classes), analyze its scale and diversity in comparison to other large-scale video datasets for action recognition, and report results of several baseline models addressing separately, and jointly, three modalities: spatial, temporal and auditory. The Moments in Time dataset, designed to have a large coverage and diversity of events in both visual and auditory modalities, can serve as a new challenge to develop models that scale to the level of complexity and abstract reasoning that a human processes on a daily basis." +9c88c2357abcd58cc330179c1965fe0a8c067ebc,1709.00029,10.1109/JSTARS.2019.2918242,2017,EuroSAT: A Novel Dataset and Deep Learning Benchmark for Land Use and Land Cover Classification,P. Helber,4,IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing,3040,"In this paper, we present a patch-based land use and land cover classification approach using Sentinel-2 satellite images. The Sentinel-2 satellite images are openly and freely accessible, and are provided in the earth observation program Copernicus. We present a novel dataset, based on these images that covers 13 spectral bands and is comprised of ten classes with a total of 27 000 labeled and geo-referenced images. Benchmarks are provided for this novel dataset with its spectral bands using state-of-the-art deep convolutional neural networks. An overall classification accuracy of 98.57% was achieved with the proposed novel dataset. The resulting classification system opens a gate toward a number of earth observation applications. We demonstrate how this classification system can be used for detecting land use and land cover changes, and how it can assist in improving geographical maps. The geo-referenced dataset EuroSAT is made publicly available at https://github.com/phelber/eurosat." +135bafc83e9a73c88e759f98a28edfdb5c02f81d,1707.09457,10.18653/v1/D17-1323,2017,Men Also Like Shopping: Reducing Gender Bias Amplification using Corpus-level Constraints,Jieyu Zhao,5,Conference on Empirical Methods in Natural Language Processing,1138,"Language is increasingly being used to de-fine rich visual recognition problems with supporting image collections sourced from the web. Structured prediction models are used in these tasks to take advantage of correlations between co-occurring labels and visual input but risk inadvertently encoding social biases found in web corpora. In this work, we study data and models associated with multilabel object classification and visual semantic role labeling. We find that (a) datasets for these tasks contain significant gender bias and (b) models trained on these datasets further amplify existing bias. For example, the activity cooking is over 33% more likely to involve females than males in a training set, and a trained model further amplifies the disparity to 68% at test time. We propose to inject corpus-level constraints for calibrating existing structured prediction models and design an algorithm based on Lagrangian relaxation for collective inference. Our method results in almost no performance loss for the underlying recognition task but decreases the magnitude of bias amplification by 47.5% and 40.5% for multilabel classification and visual semantic role labeling, respectively。" +2a5667702b0f1ff77dde8fb3e2e10d4e05e8de9d,,10.1109/CVPR.2017.544,2017,Scene Parsing through ADE20K Dataset,Bolei Zhou,6,Computer Vision and Pattern Recognition,4202,"Scene parsing, or recognizing and segmenting objects and stuff in an image, is one of the key problems in computer vision. Despite the communitys efforts in data collection, there are still few image datasets covering a wide range of scenes and object categories with dense and detailed annotations for scene parsing. In this paper, we introduce and analyze the ADE20K dataset, spanning diverse annotations of scenes, objects, parts of objects, and in some cases even parts of parts. A scene parsing benchmark is built upon the ADE20K with 150 object and stuff classes included. Several segmentation baseline models are evaluated on the benchmark. A novel network design called Cascade Segmentation Module is proposed to parse a scene into stuff, objects, and object parts in a cascade and improve over the baselines. We further show that the trained scene parsing networks can lead to applications such as image content removal and scene synthesis1." +8760bc7631c0cb04e7138254e9fd6451b7def8ca,1707.02968,10.1109/ICCV.2017.97,2017,Revisiting Unreasonable Effectiveness of Data in Deep Learning Era,Chen Sun,4,IEEE International Conference on Computer Vision,2814,"The success of deep learning in vision can be attributed to: (a) models with high capacity; (b) increased computational power; and (c) availability of large-scale labeled data. Since 2012, there have been significant advances in representation capabilities of the models and computational capabilities of GPUs. But the size of the biggest dataset has surprisingly remained constant. What will happen if we increase the dataset size by 10 × or 100 × ? This paper takes a step towards clearing the clouds of mystery surrounding the relationship between ‘enormous data’ and visual deep learning. By exploiting the JFT-300M dataset which has more than 375M noisy labels for 300M images, we investigate how the performance of current vision tasks would change if this data was used for representation learning. Our paper delivers some surprising (and some expected) findings. First, we find that the performance on vision tasks increases logarithmically based on volume of training data size. Second, we show that representation learning (or pre-training) still holds a lot of promise. One can improve performance on many vision tasks by just training a better base model. Finally, as expected, we present new state-of-the-art results for different vision tasks including image classification, object detection, semantic segmentation and human pose estimation. Our sincere hope is that this inspires vision community to not undervalue the data and develop collective efforts in building larger datasets." +d65ce2b8300541414bfe51d03906fca72e93523c,1706.04599,,2017,On Calibration of Modern Neural Networks,Chuan Guo,4,International Conference on Machine Learning,10796,"Confidence calibration -- the problem of predicting probability estimates representative of the true correctness likelihood -- is important for classification models in many applications. We discover that modern neural networks, unlike those from a decade ago, are poorly calibrated. Through extensive experiments, we observe that depth, width, weight decay, and Batch Normalization are important factors influencing calibration. We evaluate the performance of various post-processing calibration methods on state-of-the-art architectures with image and document classification datasets. Our analysis and experiments not only offer insights into neural network learning, but also provide a simple and straightforward recipe for practical settings: on most datasets, temperature scaling -- a single-parameter variant of Platt Scaling -- is surprisingly effective at calibrating predictions." +86e1bdbfd13b9ed137e4c4b8b459a3980eb257f6,1705.06950,,2017,The Kinetics Human Action Video Dataset,W. Kay,12,arXiv.org,4670,"We describe the DeepMind Kinetics human action video dataset. The dataset contains 400 human action classes, with at least 400 video clips for each action. Each clip lasts around 10s and is taken from a different YouTube video. The actions are human focussed and cover a broad range of classes including human-object interactions such as playing instruments, as well as human-human interactions such as shaking hands. We describe the statistics of the dataset, how it was collected, and give some baseline performance figures for neural network architectures trained and tested for human action classification on this dataset. We also carry out a preliminary analysis of whether imbalance in the dataset leads to bias in the classifiers." +f302e136c41db5de1d624412f68c9174cf7ae8be,1703.01365,,2017,Axiomatic Attribution for Deep Networks,Mukund Sundararajan,3,International Conference on Machine Learning,9111,"We study the problem of attributing the prediction of a deep network to its input features, a problem previously studied by several other works. We identify two fundamental axioms— Sensitivity and Implementation Invariance that attribution methods ought to satisfy. We show that they are not satisfied by most known attribution methods, which we consider to be a fundamental weakness of those methods. We use the axioms to guide the design of a new attribution method called Integrated Gradients. Our method requires no modification to the original network and is extremely simple to implement; it just needs a few calls to the standard gradient operator. We apply this method to a couple of image models, a couple of text models and a chemistry model, demonstrating its ability to debug networks, to extract rules from a network, and to enable users to engage with models better." +179765729fc1e269393617795507607c29a66a8e,1703.00121,10.1109/JPROC.2017.2675998,2017,Remote Sensing Image Scene Classification: Benchmark and State of the Art,Gong Cheng,3,Proceedings of the IEEE,2991,"Remote sensing image scene classification plays an important role in a wide range of applications and hence has been receiving remarkable attention. During the past years, significant efforts have been made to develop various data sets or present a variety of approaches for scene classification from remote sensing images. However, a systematic review of the literature concerning data sets and methods for scene classification is still lacking. In addition, almost all existing data sets have a number of limitations, including the small scale of scene classes and the image numbers, the lack of image variations and diversity, and the saturation of accuracy. These limitations severely limit the development of new approaches especially deep learning-based methods. This paper first provides a comprehensive review of the recent progress. Then, we propose a large-scale data set, termed “NWPU-RESISC45,” which is a publicly available benchmark for REmote Sensing Image Scene Classification (RESISC), created by Northwestern Polytechnical University (NWPU). This data set contains 31 500 images, covering 45 scene classes with 700 images in each class. The proposed NWPU-RESISC45 1) is large-scale on the scene classes and the total image number; 2) holds big variations in translation, spatial resolution, viewpoint, object pose, illumination, background, and occlusion; and 3) has high within-class diversity and between-class similarity. The creation of this data set will enable the community to develop and evaluate various data-driven algorithms. Finally, several representative methods are evaluated using the proposed data set, and the results are reported as a useful baseline for future research." +03eb382e04cca8cca743f7799070869954f1402a,1612.06890,10.1109/CVPR.2017.215,2016,CLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoning,Justin Johnson,6,Computer Vision and Pattern Recognition,3096,"When building artificial intelligence systems that can reason and answer questions about visual data, we need diagnostic tests to analyze our progress and discover short-comings. Existing benchmarks for visual question answering can help, but have strong biases that models can exploit to correctly answer questions without reasoning. They also conflate multiple sources of error, making it hard to pinpoint model weaknesses. We present a diagnostic dataset that tests a range of visual reasoning abilities. It contains minimal biases and has detailed annotations describing the kind of reasoning each question requires. We use this dataset to analyze a variety of modern visual reasoning systems, providing novel insights into their abilities and limitations." +5fb808edfaae574ee7f4553ad7090ca1fed9b585,1610.08452,10.1145/3038912.3052660,2016,Fairness Beyond Disparate Treatment & Disparate Impact: Learning Classification without Disparate Mistreatment,Muhammad Bilal Zafar,4,The Web Conference,1366,"Automated data-driven decision making systems are increasingly being used to assist, or even replace humans in many settings. These systems function by learning from historical decisions, often taken by humans. In order to maximize the utility of these systems (or, classifiers), their training involves minimizing the errors (or, misclassifications) over the given historical data. However, it is quite possible that the optimally trained classifier makes decisions for people belonging to different social groups with different misclassification rates (e.g., misclassification rates for females are higher than for males), thereby placing these groups at an unfair disadvantage. To account for and avoid such unfairness, in this paper, we introduce a new notion of unfairness, disparate mistreatment, which is defined in terms of misclassification rates. We then propose intuitive measures of disparate mistreatment for decision boundary-based classifiers, which can be easily incorporated into their formulation as convex-concave constraints. Experiments on synthetic as well as real world datasets show that our methodology is effective at avoiding disparate mistreatment, often at a small cost in terms of accuracy." +5966d7c7f60898d610812e24c64d4d57855ad86a,1608.07187,10.1126/science.aal4230,2016,Semantics derived automatically from language corpora contain human-like biases,Aylin Caliskan,3,Science,3388,"Machines learn what people know implicitly AlphaGo has demonstrated that a machine can learn how to do things that people spend many years of concentrated study learning, and it can rapidly learn how to do them better than any human can. Caliskan et al. now show that machines can learn word associations from written texts and that these associations mirror those learned by humans, as measured by the Implicit Association Test (IAT) (see the Perspective by Greenwald). Why does this matter? Because the IAT has predictive value in uncovering the association between concepts, such as pleasantness and flowers or unpleasantness and insects. It can also tease out attitudes and beliefs—for example, associations between female names and family or male names and career. Such biases may not be expressed explicitly, yet they can prove influential in behavior. Science, this issue p. 183; see also p. 133 Computers can learn which words go together more or less often and can thus mimic human performance on a test of implicit bias. Machine learning is a means to derive artificial intelligence by discovering patterns in existing data. Here, we show that applying machine learning to ordinary human language results in human-like semantic biases. We replicated a spectrum of known biases, as measured by the Implicit Association Test, using a widely used, purely statistical machine-learning model trained on a standard corpus of text from the World Wide Web. Our results indicate that text corpora contain recoverable and accurate imprints of our historic biases, whether morally neutral as toward insects or flowers, problematic as toward race or gender, or even simply veridical, reflecting the status quo distribution of gender with respect to careers or first names. Our methods hold promise for identifying and addressing sources of bias in culture, including technology." +b022f2a277a4bf5f42382e86e4380b96340b9e86,1608.03983,,2016,SGDR: Stochastic Gradient Descent with Warm Restarts,I. Loshchilov,2,International Conference on Learning Representations,11715,"Restart techniques are common in gradient-free optimization to deal with multimodal functions. Partial warm restarts are also gaining popularity in gradient-based optimization to improve the rate of convergence in accelerated gradient schemes to deal with ill-conditioned functions. In this paper, we propose a simple warm restart technique for stochastic gradient descent to improve its anytime performance when training deep neural networks. We empirically study its performance on the CIFAR-10 and CIFAR-100 datasets, where we demonstrate new state-of-the-art results at 3.14% and 16.21%, respectively. We also demonstrate its advantages on a dataset of EEG recordings and on a downsampled version of the ImageNet dataset. Our source code is available at this https URL" +97fb4e3d45bb098e27e0071448b6152217bd35a5,1607.06450,,2016,Layer Normalization,Jimmy Ba,3,arXiv.org,13427,"Training state-of-the-art, deep neural networks is computationally expensive. One way to reduce the training time is to normalize the activities of the neurons. A recently introduced technique called batch normalization uses the distribution of the summed input to a neuron over a mini-batch of training cases to compute a mean and variance which are then used to normalize the summed input to that neuron on each training case. This significantly reduces the training time in feed-forward neural networks. However, the effect of batch normalization is dependent on the mini-batch size and it is not obvious how to apply it to recurrent neural networks. In this paper, we transpose batch normalization into layer normalization by computing the mean and variance used for normalization from all of the summed inputs to the neurons in a layer on a single training case. Like batch normalization, we also give each neuron its own adaptive bias and gain which are applied after the normalization but before the non-linearity. Unlike batch normalization, layer normalization performs exactly the same computation at training and test times. It is also straightforward to apply to recurrent neural networks by computing the normalization statistics separately at each time step. Layer normalization is very effective at stabilizing the hidden state dynamics in recurrent networks. Empirically, we show that layer normalization can substantially reduce the training time compared with previously published techniques." +67c08e2b8b918a61dcbd0d4c63a74b89b833d259,,10.1038/srep27988,2016,Multi-class texture analysis in colorectal cancer histology,Jakob Nikolas Kather,8,Scientific Reports,611,"Automatic recognition of different tissue types in histological images is an essential part in the digital pathology toolbox. Texture analysis is commonly used to address this problem; mainly in the context of estimating the tumour/stroma ratio on histological samples. However, although histological images typically contain more than two tissue types, only few studies have addressed the multi-class problem. For colorectal cancer, one of the most prevalent tumour types, there are in fact no published results on multiclass texture separation. In this paper we present a new dataset of 5,000 histological images of human colorectal cancer including eight different types of tissue. We used this set to assess the classification performance of a wide range of texture descriptors and classifiers. As a result, we found an optimal classification strategy that markedly outperformed traditional methods, improving the state of the art for tumour-stroma separation from 96.9% to 98.6% accuracy and setting a new standard for multiclass tissue separation (87.4% accuracy for eight classes). We make our dataset of histological images publicly available under a Creative Commons license and encourage other researchers to use it as a benchmark for their studies." +0c908739fbff75f03469d13d4a1a07de3414ee19,1503.02531,,2015,Distilling the Knowledge in a Neural Network,Geoffrey E. Hinton,3,arXiv.org,27151,"A very simple way to improve the performance of almost any machine learning algorithm is to train many different models on the same data and then to average their predictions. Unfortunately, making predictions using a whole ensemble of models is cumbersome and may be too computationally expensive to allow deployment to a large number of users, especially if the individual models are large neural nets. Caruana and his collaborators have shown that it is possible to compress the knowledge in an ensemble into a single model which is much easier to deploy and we develop this approach further using a different compression technique. We achieve some surprising results on MNIST and we show that we can significantly improve the acoustic model of a heavily used commercial system by distilling the knowledge in an ensemble of models into a single model. We also introduce a new type of ensemble composed of one or more full models and many specialist models which learn to distinguish fine-grained classes that the full models confuse. Unlike a mixture of experts, these specialist models can be trained rapidly and in parallel." +8d115c3b2ee80e0754360a154a9369bc1658b607,,10.1609/aaai.v29i1.9602,2015,Obtaining Well Calibrated Probabilities Using Bayesian Binning,Mahdi Pakdaman Naeini,3,AAAI Conference on Artificial Intelligence,2217,"Learning probabilistic predictive models that are well calibrated is critical for many prediction and decision-making tasks in artificial intelligence. In this paper we present a new non-parametric calibration method called Bayesian Binning into Quantiles (BBQ) which addresses key limitations of existing calibration methods. The method post processes the output of a binary classification algorithm; thus, it can be readily combined with many existing classification algorithms. The method is computationally tractable, and empirically accurate, as evidenced by the set of experiments reported here on both real and simulated datasets." +a6cb366736791bcccc5c8639de5a8f9636bf87e8,1412.6980,,2014,Adam: A Method for Stochastic Optimization,Diederik P. Kingma,2,International Conference on Learning Representations,173229,"We introduce Adam, an algorithm for first-order gradient-based optimization of stochastic objective functions, based on adaptive estimates of lower-order moments. The method is straightforward to implement, is computationally efficient, has little memory requirements, is invariant to diagonal rescaling of the gradients, and is well suited for problems that are large in terms of data and/or parameters. The method is also appropriate for non-stationary objectives and problems with very noisy and/or sparse gradients. The hyper-parameters have intuitive interpretations and typically require little tuning. Some connections to related algorithms, on which Adam was inspired, are discussed. We also analyze the theoretical convergence properties of the algorithm and provide a regret bound on the convergence rate that is comparable to the best known results under the online convex optimization framework. Empirical results demonstrate that Adam works well in practice and compares favorably to other stochastic optimization methods. Finally, we discuss AdaMax, a variant of Adam based on the infinity norm." +6424b69f3ff4d35249c0bb7ef912fbc2c86f4ff4,1411.7766,10.1109/ICCV.2015.425,2014,Deep Learning Face Attributes in the Wild,Zi-Wei Liu,4,IEEE International Conference on Computer Vision,10119,"Predicting face attributes in the wild is challenging due to complex face variations. We propose a novel deep learning framework for attribute prediction in the wild. It cascades two CNNs, LNet and ANet, which are fine-tuned jointly with attribute tags, but pre-trained differently. LNet is pre-trained by massive general object categories for face localization, while ANet is pre-trained by massive face identities for attribute prediction. This framework not only outperforms the state-of-the-art with a large margin, but also reveals valuable facts on learning face representation. (1) It shows how the performances of face localization (LNet) and attribute prediction (ANet) can be improved by different pre-training strategies. (2) It reveals that although the filters of LNet are fine-tuned only with image-level attribute tags, their response maps over entire images have strong indication of face locations. This fact enables training LNet for face localization with only image-level annotations, but without face bounding boxes or landmarks, which are required by all attribute recognition works. (3) It also demonstrates that the high-level hidden neurons of ANet automatically discover semantic concepts after pre-training with massive face identities, and such concepts are significantly enriched after fine-tuning with attribute tags. Each attribute can be well explained with a sparse linear combination of these concepts." +e15cf50aa89fee8535703b9f9512fca5bfc43327,1409.4842,10.1109/CVPR.2015.7298594,2014,Going deeper with convolutions,C. Szegedy,9,Computer Vision and Pattern Recognition,48406,"We propose a deep convolutional neural network architecture codenamed Inception that achieves the new state of the art for classification and detection in the ImageNet Large-Scale Visual Recognition Challenge 2014 (ILSVRC14). The main hallmark of this architecture is the improved utilization of the computing resources inside the network. By a carefully crafted design, we increased the depth and width of the network while keeping the computational budget constant. To optimize quality, the architectural decisions were based on the Hebbian principle and the intuition of multi-scale processing. One particular incarnation used in our submission for ILSVRC14 is called GoogLeNet, a 22 layers deep network, the quality of which is assessed in the context of classification and detection." +3419ccd5c94d301ee08d716d037f0c3c6a62e78e,,10.1109/CVPR.2014.119,2014,The Role of Context for Object Detection and Semantic Segmentation in the Wild,Roozbeh Mottaghi,8,2014 IEEE Conference on Computer Vision and Pattern Recognition,1699, +dd2cf76ae78a3262a094ac865aa9f60c55472c5d,1406.2283,,2014,Depth Map Prediction from a Single Image using a Multi-Scale Deep Network,D. Eigen,3,Neural Information Processing Systems,4943,"Predicting depth is an essential component in understanding the 3D geometry of a scene. While for stereo images local correspondence suffices for estimation, finding depth relations from a single image is less straightforward, requiring integration of both global and local information from various cues. Moreover, the task is inherently ambiguous, with a large source of uncertainty coming from the overall scale. In this paper, we present a new method that addresses this task by employing two deep network stacks: one that makes a coarse global prediction based on the entire image, and another that refines this prediction locally. We also apply a scale-invariant error to help measure depth relations rather than scale. By leveraging the raw datasets as large sources of training data, our method achieves state-of-the-art results on both NYU Depth and KITTI, and matches detailed depth boundaries without the need for superpixelation." +a83cec6a91701bd8500f8c43ad731d4353c71d55,,10.1109/ICCVW.2013.77,2013,3D Object Representations for Fine-Grained Categorization,J. Krause,4,2013 IEEE International Conference on Computer Vision Workshops,5137, +18c125ce0f64e85577f7d30132cf0e92ec664bf4,1311.3618,10.1109/CVPR.2014.461,2013,Describing Textures in the Wild,M. Cimpoi,5,2014 IEEE Conference on Computer Vision and Pattern Recognition,3905,"Patterns and textures are key characteristics of many natural objects: a shirt can be striped, the wings of a butterfly can be veined, and the skin of an animal can be scaly. Aiming at supporting this dimension in image understanding, we address the problem of describing textures with semantic attributes. We identify a vocabulary of forty-seven texture terms and use them to describe a large dataset of patterns collected ""in the wild"". The resulting Describable Textures Dataset (DTD) is a basis to seek the best representation for recognizing describable texture attributes in images. We port from object recognition to texture recognition the Improved Fisher Vector (IFV) and Deep Convolutional-network Activation Features (DeCAF), and show that surprisingly, they both outperform specialized texture descriptors not only on our problem, but also in established material recognition datasets. We also show that our describable attributes are excellent texture descriptors, transferring between datasets and tasks, in particular, combined with IFV and DeCAF, they significantly outperform the state-of-the-art by more than 10% on both FMD and KTH-TIPS-2b benchmarks. We also demonstrate that they produce intuitive descriptions of materials and Internet images." +79b949d9b35c3f51dd20fb5c746cc81fc87147eb,,10.1177/0278364913491297,2013,Vision meets robotics: The KITTI dataset,Andreas Geiger,4,Int. J. Robotics Res.,10993, +c069629a51f6c1c301eb20ed77bc6b586c24ce32,,,2011,The Caltech-UCSD Birds-200-2011 Dataset,Catherine Wah,5,,5372, +adaa0523a5c9d5f92aa2009a51226391d8e62380,1104.3913,10.1145/2090236.2090255,2011,Fairness through awareness,C. Dwork,5,Information Technology Convergence and Services,4766,"We study fairness in classification, where individuals are classified, e.g., admitted to a university, and the goal is to prevent discrimination against individuals based on their membership in some group, while maintaining utility for the classifier (the university). The main conceptual contribution of this paper is a framework for fair classification comprising (1) a (hypothetical) task-specific metric for determining the degree to which individuals are similar with respect to the classification task at hand; (2) an algorithm for maximizing utility subject to the fairness constraint, that similar individuals are treated similarly. We also present an adaptation of our approach to achieve the complementary goal of ""fair affirmative action,"" which guarantees statistical parity (i.e., the demographics of the set of individuals receiving any classification are the same as the demographics of the underlying population), while treating similar individuals as similarly as possible. Finally, we discuss the relationship of fairness to privacy: when fairness implies privacy, and how tools developed in the context of differential privacy may be applied to fairness." +75347fcd6adb2b1c5de7706713c1cc41f7b58f36,,10.1145/1869790.1869829,2010,Bag-of-visual-words and spatial extensions for land-use classification,Yi Yang,2,ACM SIGSPATIAL International Workshop on Advances in Geographic Information Systems,3230, +a25fbcbbae1e8f79c4360d26aa11a3abf1a11972,,10.1109/TKDE.2009.191,2010,A Survey on Transfer Learning,Sinno Jialin Pan,2,IEEE Transactions on Knowledge and Data Engineering,24346,"A major assumption in many machine learning and data mining algorithms is that the training and future data must be in the same feature space and have the same distribution. However, in many real-world applications, this assumption may not hold. For example, we sometimes have a classification task in one domain of interest, but we only have sufficient training data in another domain of interest, where the latter data may be in a different feature space or follow a different data distribution. In such cases, knowledge transfer, if done successfully, would greatly improve the performance of learning by avoiding much expensive data-labeling efforts. In recent years, transfer learning has emerged as a new learning framework to address this problem. This survey focuses on categorizing and reviewing the current progress on transfer learning for classification, regression, and clustering problems. In this survey, we discuss the relationship between transfer learning and other related machine learning techniques such as domain adaptation, multitask learning and sample selection bias, as well as covariate shift. We also explore some potential future issues in transfer learning research." +908091b4a8757c3b2f7d9cfa2c4f616ee12c5157,,10.1109/CVPR.2010.5539970,2010,SUN database: Large-scale scene recognition from abbey to zoo,Jian-Xiong Xiao,5,2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition,4547,"Scene categorization is a fundamental problem in computer vision. However, scene understanding research has been constrained by the limited scope of currently-used databases which do not capture the full variety of scene categories. Whereas standard databases for object categorization contain hundreds of different classes of objects, the largest available dataset of scene categories contains only 15 classes. In this paper we propose the extensive Scene UNderstanding (SUN) database that contains 899 categories and 130,519 images. We use 397 well-sampled categories to evaluate numerous state-of-the-art algorithms for scene recognition and establish new bounds of performance. We measure human scene classification performance on the SUN database and compare this with computational methods. Additionally, we study a finer-grained scene representation to detect scenes embedded inside of larger scenes." +bcd842c0e6e731f347523d774b089cdf21d9e8f1,,10.5555/1756006.1859904,2010,On the Foundations of Noise-free Selective Classification,Ran El-Yaniv,2,Journal of machine learning research,747, +d2c733e34d48784a37d717fe43d9e93277a8c53e,,10.1109/CVPR.2009.5206848,2009,ImageNet: A large-scale hierarchical image database,Jia Deng,6,2009 IEEE Conference on Computer Vision and Pattern Recognition,77302, +02b28f3b71138a06e40dbd614abf8568420ae183,,10.1109/ICVGIP.2008.47,2008,Automated Flower Classification over a Large Number of Classes,M. Nilsback,2,"2008 Sixth Indian Conference on Computer Vision, Graphics & Image Processing",5235, +f354310098e09c1e1dc88758fca36767fd9d084d,,10.1109/CVPR.2004.144,2004,Learning methods for generic object recognition with invariance to pose and lighting,Yann LeCun,3,"Proceedings of the 2004 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2004. CVPR 2004.",1600, +47a9ab6f97ab05fcd49aaf2864c97538b55e6268,,10.1137/0916069,1995,A Limited Memory Algorithm for Bound Constrained Optimization,R. Byrd,4,SIAM Journal on Scientific Computing,6989, +db95b0785118db34b553e6b5f137ea2bee5a0639,,10.1136/bmj.4.5674.53-c,1969,Care of aged doctors,P. Gooch,1,,1856, +f40aeae3e522ada1f6a9f326841b01ef5c8657b6,,,2022,Unifying Language Learning Paradigms,Yi Tay,9,arXiv.org,140, +919441bd618c2aff89dfa5affdb62cf6e0c4e293,1911.11132,,2022,Scaling Out-of-Distribution Detection for Real-World Settings,Dan Hendrycks,7,International Conference on Machine Learning,742,"Detecting out-of-distribution examples is important for safety-critical machine learning applications such as medical screening and self-driving cars. However, existing research mainly focuses on simple small-scale settings. To set the stage for more realistic out-of-distribution detection, we depart from small-scale settings and explore large-scale multiclass and multi-label settings with high-resolution images and hundreds of classes. To make future work in real-world settings possible, we also create a new benchmark for anomaly segmentation by introducing the Combined Anomalous Object Segmentation benchmark. Our novel benchmark combines two datasets for anomaly segmentation that incorporate both realism and anomaly diversity. Using both real images and those from a simulated driving environment, we ensure the background context and a wide variety of anomalous objects are naturally integrated, unlike before. We conduct extensive experiments in these more realistic settings for out-of-distribution detection and find that a surprisingly simple detector based on the maximum logit outperforms prior methods in all the large-scale multi-class, multi-label, and segmentation tasks we consider, establishing a new baseline for future work. These results, along with our new anomaly segmentation benchmark, open the door to future research in out-of-distribution detection." +,,,2022,"2022) and because a distribution normalised across the classes is required by several of the metrics employed (e.g., ECE)",,0,,0, +,,,2022,"2022a), we resize the trained embedding layer from the default patch-size of 14 to a patch-size",,0,,0, +,,,2022,2022) show a trade-off between the accuracy of latest ImageNet classifiers and their inherent ability to capture perceptual similarity,,0,,0, +,,,2021,"corruptions and over their 5 different intensity levels. OOD detection (Hendrycks et al., 2019",,0,,0, +,,,2021,"Dense Prediction Transformer. We largely follow the design of (Ranftl et al., 2021), using four reassemble and fusion blocks that processes the 16× 16 ViT feature map at",,0,,0, +,,,2021,GPT-J-6B: A 6 Billion Parameter Autoregressive Language Model,,0,github,0, +,,,2020,Details about the evaluation We start by providing some details about the datasets and the different evaluation protocols based on,,0,Djolonga et al,0, +,,,2020,"2021), with a review fraction",,0,,0, +,,,2020,URL https://github.com/google-research/ robustness_metrics,,0,"Robustness Metrics,",0, +,,,2020,Flax: A neural network library and ecosystem for JAX,,0,,0, +639174f32a71ecfe9041ad05ff30eb39bd4977bf,,,2019,ObjectNet: A large-scale bias-controlled dataset for pushing the limits of object recognition models,Andrei Barbu,8,Neural Information Processing Systems,713, +,,,2019,"In this task, we try to classify whether a given test point belongs to the in-distribution dataset (in our case, ImageNet) or an out-of-distribution dataset (following Hendrycks et al",,0,"OOD detection (Hendrycks et al.,",0, +,,,2018,we assess the ability of ViT-22B to capture perceptual similarity using 48 intermediate representations. The perceptual score of ViT-22B (64.9) is much lower than all other models,,0,ImageNet. ViT-22B fine-tuned on ImageNet 64 × 64 achieves 84.2 accuracy on ImageNet 64 × 64 which is 16% better than the best models trained directly on ImageNet. As done in,0, +,,,2018,"JAX: composable transformations of Python+NumPy programs, 2018",,0,URL http://github.com/google/jax,0, +,,,2017,dSprites: Disentanglement testing sprites dataset,,0,,0, +,,,2015,Kaggle and EyePacs,,0,Kaggle diabetic retinopathy,0, +02227c94dd41fe0b439e050d377b0beb5d427cda,,,2011,Reading Digits in Natural Images with Unsupervised Feature Learning,Y. Netzer,6,,8369, +5d90f06bb70a0a3dced62413346235c02b1aa086,,,2009,Learning Multiple Layers of Features from Tiny Images,A. Krizhevsky,1,,45138, +,,,2009,Evaluation Dataset We evaluate the ViT-22B on a wide variety of tasks and report the results on each individual tasks and datasets,,0,ACM International Conference on Architectural Support for Programming Languages and Operating Systems,0, +,,,2004,ImageNet-ReaL-H,,0,Waymo Open real-world driving dataset,0, +34cee5526c020ab0abfe71e80d219022441a35d9,,10.1038/SJ/JORS/0420101,1991,The Many Faces of OR,S. Gass,1,,391, +3a9b175324ba11bc0e16c0633912d897b2fac4e2,,,,International Journal of Computer Vision manuscript No. (will be inserted by the editor) The PASCAL Visual Object Classes (VOC) Challenge,,0,,16549, +,,,,ViT-22B with the newly proposed Mean Pool distance function,,0,,0, +,,,,"Intriguing Properties of Transformer Training Instabilities, 2023",,0,To appear,0, diff --git a/02-literature/search/results/snowball-back-wortsman2024.csv b/02-literature/search/results/snowball-back-wortsman2024.csv new file mode 100644 index 0000000..473080f --- /dev/null +++ b/02-literature/search/results/snowball-back-wortsman2024.csv @@ -0,0 +1,57 @@ +s2_id,arxiv,doi,year,title,first_author,n_authors,venue,citations,abstract +7368c3cdf7cbed194e96dc4da53ed61f185e3d82,2309.08586,10.48550/arXiv.2309.08586,2023,Replacing softmax with ReLU in Vision Transformers,Mitchell Wortsman,4,arXiv.org,77,"Previous research observed accuracy degradation when replacing the attention softmax with a point-wise activation such as ReLU. In the context of vision transformers, we find that this degradation is mitigated when dividing by sequence length. Our experiments training small to large vision transformers on ImageNet-21k indicate that ReLU-attention can approach or match the performance of softmax-attention in terms of scaling behavior as a function of compute." +b3c64a046d6cfc0f55a2aebc5176bbab69a7e021,2304.13013,10.48550/arXiv.2304.13013,2023,Stable and low-precision training for large-scale vision-language models,Mitchell Wortsman,6,Neural Information Processing Systems,106,"We introduce new methods for 1) accelerating and 2) stabilizing training for large language-vision models. 1) For acceleration, we introduce SwitchBack, a linear layer for int8 quantized training which provides a speed-up of 13-25% while matching the performance of bfloat16 training within 0.1 percentage points for the 1B parameter CLIP ViT-Huge -- the largest int8 training to date. Our main focus is int8 as GPU support for float8 is rare, though we also analyze float8 training through simulation. While SwitchBack proves effective for float8, we show that standard techniques are also successful if the network is trained and initialized so that large feature magnitudes are discouraged, which we accomplish via layer-scale initialized with zeros. 2) For stability, we analyze loss spikes and find they consistently occur 1-8 iterations after the squared gradients become under-estimated by their AdamW second moment estimator. As a result, we recommend an AdamW-Adafactor hybrid which avoids loss spikes when training a CLIP ViT-Huge model and outperforms gradient clipping at the scales we test." +88c94b11bd18161d027a28dd758b58698063e029,2304.09871,10.48550/arXiv.2304.09871,2023,A Theory on Adam Instability in Large-Scale Machine Learning,Igor Molybog,17,arXiv.org,60,"We present a theory for the previously unexplained divergent behavior noticed in the training of large language models. We argue that the phenomenon is an artifact of the dominant optimization algorithm used for training, called Adam. We observe that Adam can enter a state in which the parameter update vector has a relatively large norm and is essentially uncorrelated with the direction of descent on the training loss landscape, leading to divergence. This artifact is more likely to be observed in the training of a deep model with a large batch size, which is the typical setting of large-scale language model training. To argue the theory, we present observations from the training runs of the language models of different scales: 7 billion, 30 billion, 65 billion, and 546 billion parameters." +fe1be27f0f3ad3399ae5aea1e5d3eb06251a64af,2304.03408,10.1088/1742-5468/ad642b,2023,Dynamics of finite width Kernel and prediction fluctuations in mean field neural networks,Blake Bordelon,2,Neural Information Processing Systems,57,"We analyze the dynamics of finite width effects in wide but finite feature learning neural networks. Starting from a dynamical mean field theory description of infinite width deep neural network kernel and prediction dynamics, we provide a characterization of the O(1/width) fluctuations of the dynamical mean field theory order parameters over random initializations of the network weights. Our results, while perturbative in width, unlike prior analyses, are non-perturbative in the strength of feature learning. We find that once the mean field/µP parameterization is adopted, the leading finite size effect on the dynamics is to introduce initialization variance in the predictions and feature kernels of the networks. In the lazy limit of network training, all kernels are random but static in time and the prediction variance has a universal form. However, in the rich, feature learning regime, the fluctuations of the kernels and predictions are dynamically coupled with a variance that can be computed self-consistently. In two layer networks, we show how feature learning can dynamically reduce the variance of the final tangent kernel and final network predictions. We also show how initialization variance can slow down online learning in wide but finite networks. In deeper networks, kernel variance can dramatically accumulate through subsequent layers at large feature learning strengths, but feature learning continues to improve the signal-to-noise ratio of the feature kernels. In discrete time, we demonstrate that large learning rate phenomena such as edge of stability effects can be well captured by infinite width dynamics and that initialization variance can decrease dynamically. For convolutional neural networks trained on CIFAR-10, we empirically find significant corrections to both the bias and variance of network dynamics due to finite width." +5c326260baa5e91a6a39c4d87ebe6770ade86816,2304.02034,10.48550/arXiv.2304.02034,2023,Effective Theory of Transformers at Initialization,Emily Dinan,3,arXiv.org,21,"We perform an effective-theory analysis of forward-backward signal propagation in wide and deep Transformers, i.e., residual neural networks with multi-head self-attention blocks and multilayer perceptron blocks. This analysis suggests particular width scalings of initialization and training hyperparameters for these models. We then take up such suggestions, training Vision and Language Transformers in practical setups." +35aba190f28b5c39df333c06ca21f46bd4845eba,2303.15343,10.1109/ICCV51070.2023.01100,2023,Sigmoid Loss for Language Image Pre-Training,Xiao-Hua Zhai,4,IEEE International Conference on Computer Vision,4190,"We propose a simple pairwise sigmoid loss for imagetext pre-training. Unlike standard contrastive learning with softmax normalization, the sigmoid loss operates solely on image-text pairs and does not require a global view of the pairwise similarities for normalization. The sigmoid loss simultaneously allows further scaling up the batch size, while also performing better at smaller batch sizes. With only four TPUv4 chips, we can train a Base CLIP model at 4k batch size and a Large LiT model at 20k batch size, the latter achieves 84.5% ImageNet zero-shot accuracy in two days. This disentanglement of the batch size from the loss further allows us to study the impact of examples vs pairs and negative to positive ratio. Finally, we push the batch size to the extreme, up to one million, and find that the benefits of growing batch size quickly diminish, with a more reasonable batch size of 32k being sufficient. We hope our research motivates further explorations in improving the quality and efficiency of language-image pre-training." +385c363ea8e450f362d389f401beaeb5b42a0022,2303.06296,10.48550/arXiv.2303.06296,2023,Stabilizing Transformer Training by Preventing Attention Entropy Collapse,Shuangfei Zhai,8,International Conference on Machine Learning,221,"Training stability is of great importance to Transformers. In this work, we investigate the training dynamics of Transformers by examining the evolution of the attention layers. In particular, we track the attention entropy for each attention head during the course of training, which is a proxy for model sharpness. We identify a common pattern across different architectures and tasks, where low attention entropy is accompanied by high training instability, which can take the form of oscillating loss or divergence. We denote the pathologically low attention entropy, corresponding to highly concentrated attention scores, as $\textit{entropy collapse}$. As a remedy, we propose $\sigma$Reparam, a simple and efficient solution where we reparametrize all linear layers with spectral normalization and an additional learned scalar. We demonstrate that $\sigma$Reparam successfully prevents entropy collapse in the attention layers, promoting more stable training. Additionally, we prove a tight lower bound of the attention entropy, which decreases exponentially fast with the spectral norm of the attention logits, providing additional motivation for our approach. We conduct experiments with $\sigma$Reparam on image classification, image self-supervised learning, machine translation, speech recognition, and language modeling tasks. We show that $\sigma$Reparam provides stability and robustness with respect to the choice of hyperparameters, going so far as enabling training (a) a Vision Transformer {to competitive performance} without warmup, weight decay, layer normalization or adaptive optimizers; (b) deep architectures in machine translation and (c) speech recognition to competitive performance without warmup and adaptive optimizers. Code is available at \url{https://github.com/apple/ml-sigma-reparam}." +61e721334296ebfbbf6443b5ed9eb8c83b708c95,2302.05442,10.48550/arXiv.2302.05442,2023,Scaling Vision Transformers to 22 Billion Parameters,Mostafa Dehghani,42,International Conference on Machine Learning,972,"The scaling of Transformers has driven breakthrough capabilities for language models. At present, the largest large language models (LLMs) contain upwards of 100B parameters. Vision Transformers (ViT) have introduced the same architecture to image and video modelling, but these have not yet been successfully scaled to nearly the same degree; the largest dense ViT contains 4B parameters (Chen et al., 2022). We present a recipe for highly efficient and stable training of a 22B-parameter ViT (ViT-22B) and perform a wide variety of experiments on the resulting model. When evaluated on downstream tasks (often with a lightweight linear model on frozen features), ViT-22B demonstrates increasing performance with scale. We further observe other interesting benefits of scale, including an improved tradeoff between fairness and performance, state-of-the-art alignment to human visual perception in terms of shape/texture bias, and improved robustness. ViT-22B demonstrates the potential for""LLM-like""scaling in vision, and provides key steps towards getting there." +aed643b9307f511a2928608b7967425c7eb3557e,2302.12022,10.48550/arXiv.2302.12022,2023,DoG is SGD's Best Friend: A Parameter-Free Dynamic Step Size Schedule,Maor Ivgi,3,International Conference on Machine Learning,111,"We propose a tuning-free dynamic SGD step size formula, which we call Distance over Gradients (DoG). The DoG step sizes depend on simple empirical quantities (distance from the initial point and norms of gradients) and have no ``learning rate'' parameter. Theoretically, we show that a slight variation of the DoG formula enjoys strong parameter-free convergence guarantees for stochastic convex optimization assuming only \emph{locally bounded} stochastic gradients. Empirically, we consider a broad range of vision and language transfer learning tasks, and show that DoG's performance is close to that of SGD with tuned learning rate. We also propose a per-layer variant of DoG that generally outperforms tuned SGD, approaching the performance of tuned Adam. A PyTorch implementation is available at https://github.com/formll/dog" +e750cf66d0e7ade8e21da38df3eb2a174b027f48,2301.07733,10.48550/arXiv.2301.07733,2023,Learning-Rate-Free Learning by D-Adaptation,Aaron Defazio,2,International Conference on Machine Learning,145,"D-Adaptation is an approach to automatically setting the learning rate which asymptotically achieves the optimal rate of convergence for minimizing convex Lipschitz functions, with no back-tracking or line searches, and no additional function value or gradient evaluations per step. Our approach is the first hyper-parameter free method for this class without additional multiplicative log factors in the convergence rate. We present extensive experiments for SGD and Adam variants of our method, where the method automatically matches hand-tuned learning rates across more than a dozen diverse machine learning problems, including large-scale vision and language problems. An open-source implementation is available." +b941c4bea1a71066f6f32275641aea1efc99b21b,2210.04909,10.48550/arXiv.2210.04909,2022,Meta-Principled Family of Hyperparameter Scaling Strategies,Sho Yaida,1,arXiv.org,20,"In this note, we first derive a one-parameter family of hyperparameter scaling strategies that interpolates between the neural-tangent scaling and mean-field/maximal-update scaling. We then calculate the scalings of dynamical observables -- network outputs, neural tangent kernels, and differentials of neural tangent kernels -- for wide and deep neural networks. These calculations in turn reveal a proper way to scale depth with width such that resultant large-scale models maintain their representation-learning ability. Finally, we observe that various infinite-width limits examined in the literature correspond to the distinct corners of the interconnected web spanned by effective theories for finite-width neural networks, with their training dynamics ranging from being weakly-coupled to being strongly-coupled." +f21a88af78583bd7959b121b800eed5c1f7b7b99,2209.15594,10.48550/arXiv.2209.15594,2022,Self-Stabilization: The Implicit Bias of Gradient Descent at the Edge of Stability,Alexandru Damian,3,International Conference on Learning Representations,178,"Traditional analyses of gradient descent show that when the largest eigenvalue of the Hessian, also known as the sharpness $S(\theta)$, is bounded by $2/\eta$, training is""stable""and the training loss decreases monotonically. Recent works, however, have observed that this assumption does not hold when training modern neural networks with full batch or large batch gradient descent. Most recently, Cohen et al. (2021) observed two important phenomena. The first, dubbed progressive sharpening, is that the sharpness steadily increases throughout training until it reaches the instability cutoff $2/\eta$. The second, dubbed edge of stability, is that the sharpness hovers at $2/\eta$ for the remainder of training while the loss continues decreasing, albeit non-monotonically. We demonstrate that, far from being chaotic, the dynamics of gradient descent at the edge of stability can be captured by a cubic Taylor expansion: as the iterates diverge in direction of the top eigenvector of the Hessian due to instability, the cubic term in the local Taylor expansion of the loss function causes the curvature to decrease until stability is restored. This property, which we call self-stabilization, is a general property of gradient descent and explains its behavior at the edge of stability. A key consequence of self-stabilization is that gradient descent at the edge of stability implicitly follows projected gradient descent (PGD) under the constraint $S(\theta) \le 2/\eta$. Our analysis provides precise predictions for the loss, sharpness, and deviation from the PGD trajectory throughout training, which we verify both empirically in a number of standard settings and theoretically under mild conditions. Our analysis uncovers the mechanism for gradient descent's implicit bias towards stability." +4be7d1524edb0137599a5cc95f72844b85a52fe1,2208.07339,10.48550/arXiv.2208.07339,2022,LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale,Tim Dettmers,4,Advances in Neural Information Processing Systems 35,1380,"Large language models have been widely adopted but require significant GPU memory for inference. We develop a procedure for Int8 matrix multiplication for feed-forward and attention projection layers in transformers, which cut the memory needed for inference by half while retaining full precision performance. With our method, a 175B parameter 16/32-bit checkpoint can be loaded, converted to Int8, and used immediately without performance degradation. This is made possible by understanding and working around properties of highly systematic emergent features in transformer language models that dominate attention and transformer predictive performance. To cope with these features, we develop a two-part quantization procedure, LLM.int8(). We first use vector-wise quantization with separate normalization constants for each inner product in the matrix multiplication, to quantize most of the features. However, for the emergent outliers, we also include a new mixed-precision decomposition scheme, which isolates the outlier feature dimensions into a 16-bit matrix multiplication while still more than 99.9% of values are multiplied in 8-bit. Using LLM.int8(), we show empirically it is possible to perform inference in LLMs with up to 175B parameters without any performance degradation. This result makes such models much more accessible, for example making it possible to use OPT-175B/BLOOM on a single server with consumer GPUs. We open-source our software." +84f6ab620eb7b112e5b2ca64b305970894e679c1,2207.14484,10.48550/arXiv.2207.14484,2022,Adaptive Gradient Methods at the Edge of Stability,Jeremy M. Cohen,11,arXiv.org,105,"Very little is known about the training dynamics of adaptive gradient methods like Adam in deep learning. In this paper, we shed light on the behavior of these algorithms in the full-batch and sufficiently large batch settings. Specifically, we empirically demonstrate that during full-batch training, the maximum eigenvalue of the preconditioned Hessian typically equilibrates at a certain numerical value -- the stability threshold of a gradient descent algorithm. For Adam with step size $\eta$ and $\beta_1 = 0.9$, this stability threshold is $38/\eta$. Similar effects occur during minibatch training, especially as the batch size grows. Yet, even though adaptive methods train at the ``Adaptive Edge of Stability'' (AEoS), their behavior in this regime differs in a significant way from that of non-adaptive methods at the EoS. Whereas non-adaptive algorithms at the EoS are blocked from entering high-curvature regions of the loss landscape, adaptive gradient methods at the AEoS can keep advancing into high-curvature regions, while adapting the preconditioner to compensate. Our findings can serve as a foundation for the community's future understanding of adaptive gradient methods in deep learning." +ff53cb49cb18e71e622fce7d96692e813630e878,2206.04817,10.48550/arXiv.2206.04817,2022,The Slingshot Mechanism: An Empirical Study of Adaptive Optimizers and the Grokking Phenomenon,Vimal Thilak,6,arXiv.org,111,"The grokking phenomenon as reported by Power et al. ( arXiv:2201.02177 ) refers to a regime where a long period of overfitting is followed by a seemingly sudden transition to perfect generalization. In this paper, we attempt to reveal the underpinnings of Grokking via a series of empirical studies. Specifically, we uncover an optimization anomaly plaguing adaptive optimizers at extremely late stages of training, referred to as the Slingshot Mechanism. A prominent artifact of the Slingshot Mechanism can be measured by the cyclic phase transitions between stable and unstable training regimes, and can be easily monitored by the cyclic behavior of the norm of the last layers weights. We empirically observe that without explicit regularization, Grokking as reported in ( arXiv:2201.02177 ) almost exclusively happens at the onset of Slingshots, and is absent without it. While common and easily reproduced in more general settings, the Slingshot Mechanism does not follow from any known optimization theories that we are aware of, and can be easily overlooked without an in depth examination. Our work points to a surprising and useful inductive bias of adaptive gradient optimizers at late stages of training, calling for a revised theoretical analysis of their origin." +13a0d8bb38f739990c8cd65a44061c6534f17221,2205.01068,,2022,OPT: Open Pre-trained Transformer Language Models,Susan Zhang,19,arXiv.org,5077,"Large language models, which are often trained for hundreds of thousands of compute days, have shown remarkable capabilities for zero- and few-shot learning. Given their computational cost, these models are difficult to replicate without significant capital. For the few that are available through APIs, no access is granted to the full model weights, making them difficult to study. We present Open Pre-trained Transformers (OPT), a suite of decoder-only pre-trained transformers ranging from 125M to 175B parameters, which we aim to fully and responsibly share with interested researchers. We show that OPT-175B is comparable to GPT-3, while requiring only 1/7th the carbon footprint to develop. We are also releasing our logbook detailing the infrastructure challenges we faced, along with code for experimenting with all of the released models." +094ff971d6a8b8ff870946c9b3ce5aa173617bfb,2204.02311,,2022,PaLM: Scaling Language Modeling with Pathways,A. Chowdhery,67,Journal of machine learning research,8533,"Large language models have been shown to achieve remarkable performance across a variety of natural language tasks using few-shot learning, which drastically reduces the number of task-specific training examples needed to adapt the model to a particular application. To further our understanding of the impact of scale on few-shot learning, we trained a 540-billion parameter, densely activated, Transformer language model, which we call Pathways Language Model PaLM. We trained PaLM on 6144 TPU v4 chips using Pathways, a new ML system which enables highly efficient training across multiple TPU Pods. We demonstrate continued benefits of scaling by achieving state-of-the-art few-shot learning results on hundreds of language understanding and generation benchmarks. On a number of these tasks, PaLM 540B achieves breakthrough performance, outperforming the finetuned state-of-the-art on a suite of multi-step reasoning tasks, and outperforming average human performance on the recently released BIG-bench benchmark. A significant number of BIG-bench tasks showed discontinuous improvements from model scale, meaning that performance steeply increased as we scaled to our largest model. PaLM also has strong capabilities in multilingual tasks and source code generation, which we demonstrate on a wide array of benchmarks. We additionally provide a comprehensive analysis on bias and toxicity, and study the extent of training data memorization with respect to model scale. Finally, we discuss the ethical considerations related to large language models and discuss potential mitigation strategies." +8342b592fe238f3d230e4959b06fd10153c45db1,2203.15556,10.52202/068431-2176,2022,Training Compute-Optimal Large Language Models,Jordan Hoffmann,22,Advances in Neural Information Processing Systems 35,3975,"We investigate the optimal model size and number of tokens for training a transformer language model under a given compute budget. We find that current large language models are significantly undertrained, a consequence of the recent focus on scaling language models whilst keeping the amount of training data constant. By training over 400 language models ranging from 70 million to over 16 billion parameters on 5 to 500 billion tokens, we find that for compute-optimal training, the model size and the number of training tokens should be scaled equally: for every doubling of model size the number of training tokens should also be doubled. We test this hypothesis by training a predicted compute-optimal model, Chinchilla, that uses the same compute budget as Gopher but with 70B parameters and 4$\times$ more more data. Chinchilla uniformly and significantly outperforms Gopher (280B), GPT-3 (175B), Jurassic-1 (178B), and Megatron-Turing NLG (530B) on a large range of downstream evaluation tasks. This also means that Chinchilla uses substantially less compute for fine-tuning and inference, greatly facilitating downstream usage. As a highlight, Chinchilla reaches a state-of-the-art average accuracy of 67.5% on the MMLU benchmark, greater than a 7% improvement over Gopher." +0b0d7d87c58d41b92d907347b778032be5966f60,2203.03466,10.48550/arXiv.2203.03466,2022,Tensor Programs V: Tuning Large Neural Networks via Zero-Shot Hyperparameter Transfer,Greg Yang,10,arXiv.org,340,"Hyperparameter (HP) tuning in deep learning is an expensive process, prohibitively so for neural networks (NNs) with billions of parameters. We show that, in the recently discovered Maximal Update Parametrization (muP), many optimal HPs remain stable even as model size changes. This leads to a new HP tuning paradigm we call muTransfer: parametrize the target model in muP, tune the HP indirectly on a smaller model, and zero-shot transfer them to the full-sized model, i.e., without directly tuning the latter at all. We verify muTransfer on Transformer and ResNet. For example, 1) by transferring pretraining HPs from a model of 13M parameters, we outperform published numbers of BERT-large (350M parameters), with a total tuning cost equivalent to pretraining BERT-large once; 2) by transferring from 40M parameters, we outperform published numbers of the 6.7B GPT-3 model, with tuning cost only 7% of total pretraining cost. A Pytorch implementation of our technique can be found at github.com/microsoft/mup and installable via `pip install mup`." +dc0102a51a9d33e104a4a3808a18cf17f057228c,2202.10447,,2022,Transformer Quality in Linear Time,Weizhe Hua,4,International Conference on Machine Learning,391,"We revisit the design choices in Transformers, and propose methods to address their weaknesses in handling long sequences. First, we propose a simple layer named gated attention unit, which allows the use of a weaker single-head attention with minimal quality loss. We then propose a linear approximation method complementary to this new layer, which is accelerator-friendly and highly competitive in quality. The resulting model, named FLASH, matches the perplexity of improved Transformers over both short (512) and long (8K) context lengths, achieving training speedups of up to 4.9$\times$ on Wiki-40B and 12.1$\times$ on PG-19 for auto-regressive language modeling, and 4.8$\times$ on C4 for masked language modeling." +83474f6510e0b985595f936d233321e131966bae,2110.04369,,2021,A Loss Curvature Perspective on Training Instability in Deep Learning,J. Gilmer,9,arXiv.org,53,"In this work, we study the evolution of the loss Hessian across many classification tasks in order to understand the effect the curvature of the loss has on the training dynamics. Whereas prior work has focused on how different learning rates affect the loss Hessian observed during training, we also analyze the effects of model initialization, architectural choices, and common training heuristics such as gradient clipping and learning rate warmup. Our results demonstrate that successful model and hyperparameter choices allow the early optimization trajectory to either avoid -- or navigate out of -- regions of high curvature and into flatter regions that tolerate a higher learning rate. Our results suggest a unifying perspective on how disparate mitigation strategies for training instability ultimately address the same underlying failure mode of neural network optimization, namely poor conditioning. Inspired by the conditioning perspective, we show that learning rate warmup can improve training stability just as much as batch normalization, layer normalization, MetaInit, GradInit, and Fixup initialization." +66c10bf1f11bc1b2d92204d8f8391d087f6de1c4,2104.09864,10.1016/j.neucom.2023.127063,2021,RoFormer: Enhanced Transformer with Rotary Position Embedding,Jian-Lin Su,6,Neurocomputing,6996,"Position encoding recently has shown effective in the transformer architecture. It enables valuable supervision for dependency modeling between elements at different positions of the sequence. In this paper, we first investigate various methods to integrate positional information into the learning process of transformer-based language models. Then, we propose a novel method named Rotary Position Embedding(RoPE) to effectively leverage the positional information. Specifically, the proposed RoPE encodes the absolute position with a rotation matrix and meanwhile incorporates the explicit relative position dependency in self-attention formulation. Notably, RoPE enables valuable properties, including the flexibility of sequence length, decaying inter-token dependency with increasing relative distances, and the capability of equipping the linear self-attention with relative position encoding. Finally, we evaluate the enhanced transformer with rotary position embedding, also called RoFormer, on various long text classification benchmark datasets. Our experiments show that it consistently overcomes its alternatives. Furthermore, we provide a theoretical analysis to explain some experimental results. RoFormer is already integrated into Huggingface: \url{https://huggingface.co/docs/transformers/model_doc/roformer}." +739ceacfafb1c4eaa17509351b647c773270b3ae,2104.02057,10.1109/ICCV48922.2021.00950,2021,An Empirical Study of Training Self-Supervised Vision Transformers,Xinlei Chen,3,IEEE International Conference on Computer Vision,2549,"This paper does not describe a novel method. Instead, it studies a straightforward, incremental, yet must-know baseline given the recent progress in computer vision: self-supervised learning for Vision Transformers (ViT). While the training recipes for standard convolutional networks have been highly mature and robust, the recipes for ViT are yet to be built, especially in the self-supervised scenarios where training becomes more challenging. In this work, we go back to basics and investigate the effects of several fundamental components for training self-supervised ViT. We observe that instability is a major issue that degrades accuracy, and it can be hidden by apparently good results. We reveal that these results are indeed partial failure, and they can be improved when training is made more stable. We benchmark ViT results in MoCo v3 and several other self-supervised frameworks, with ablations in various aspects. We discuss the currently positive evidence as well as challenges and open questions. We hope that this work will provide useful data points and experience for future research." +1480446723a1aefc26ebb0e6f105c35ab52ddb42,2103.00065,,2021,Gradient Descent on Neural Networks Typically Occurs at the Edge of Stability,Jeremy M. Cohen,5,International Conference on Learning Representations,522,"We empirically demonstrate that full-batch gradient descent on neural network training objectives typically operates in a regime we call the Edge of Stability. In this regime, the maximum eigenvalue of the training loss Hessian hovers just above the numerical value $2 / \text{(step size)}$, and the training loss behaves non-monotonically over short timescales, yet consistently decreases over long timescales. Since this behavior is inconsistent with several widespread presumptions in the field of optimization, our findings raise questions as to whether these presumptions are relevant to neural network training. We hope that our findings will inspire future efforts aimed at rigorously understanding optimization at the Edge of Stability. Code is available at https://github.com/locuslab/edge-of-stability." +12b71736392209b4292471b7da0aed71ba2aa545,2101.06840,,2021,ZeRO-Offload: Democratizing Billion-Scale Model Training,Jie Ren,8,USENIX Annual Technical Conference,673,"Large-scale model training has been a playing ground for a limited few requiring complex model refactoring and access to prohibitively expensive GPU clusters. ZeRO-Offload changes the large model training landscape by making large model training accessible to nearly everyone. It can train models with over 13 billion parameters on a single GPU, a 10x increase in size compared to popular framework such as PyTorch, and it does so without requiring any model change from the data scientists or sacrificing computational efficiency. ZeRO-Offload enables large model training by offloading data and compute to CPU. To preserve compute efficiency, it is designed to minimize the data movement to/from GPU, and reduce CPU compute time while maximizing memory savings on GPU. As a result, ZeRO-Offload can achieve 40 TFlops/GPU on a single NVIDIA V100 GPU for 10B parameter model compared to 30TF using PyTorch alone for a 1.4B parameter model, the largest that can be trained without running out of memory. ZeRO-Offload is also designed to scale on multiple-GPUs when available, offering near linear speedup on up to 128 GPUs. Additionally, it can work together with model parallelism to train models with over 70 billion parameters on a single DGX-2 box, a 4.5x increase in model size compared to using model parallelism alone. By combining compute and memory efficiency with ease-of-use, ZeRO-Offload democratizes large-scale model training making it accessible to even data scientists with access to just a single GPU." +b986e90a525dee51c8a80341416a1bc97f668ac7,,10.3136/nskkk.42.541,2021,Neural Network,Phil Kim,1,Natural Language Processing,950,"In this research, a fast, accurate, and stable system of lung cancer detection based on novel deep learning techniques is proposed. Lung cancer continues to be one of the most monumental global health concerns, which is why there is an urgent need for low-cost and non-invasive screening. Though the diagnostic methods that are most commonly in use include CTscan, X-ray etc. The interpretation by the human eye varies and errors are bound to occur. In response to this challenge, we outline a more automated approach that is based on deep learning models and can be used to classify lung pictures with high levels of accuracy. This research makes use of a large data set of lung scans categorised as normal, malignant, and benign. The first look what the data had in store threw up some correlation with picture size and what seemed to be category differences. Realizing that live feed requires constant input, each picture underwent grayscale conversion and dimensionality reduction. In order to effectively deal with the unbalanced nature of the dataset that was discovered in the study, the Synthetic Minority Oversampling Technique (SMOTE) was applied as a technique. In this presentation, three new designs were introduced: Model I, Model 2, and Model 3. Additionally, one architecture was developed with the purpose of merging the predictions of all three models. Furthermore, out of all the models created, the best model emerged as model 1 with approximately an accuracy of 84%. 7%. But the ensemble strategy which was intended to make the best of each of the models, produced an astounding 82. 5% accuracy. The specific advantages and misclassification behaviors of Model 2 and 3, although less accurate than Model 1 but are currently under evaluation for future Model ensemble improvements. The technique developed using deep learning addresses the challenges at a faster, efficient, and contactless approach to lung cancer analysis. The fact that it is capable of operating in tandem with others diagnostic instruments may help reduce diagnostic errors and enhance patient care. We have addressed this issue so that the various practitioners would be able to read this paper and we can go to the next generation of diagnostic technologies." +268d347e8a55b5eb82fb5e7d2f800e33c75ab18a,2010.11929,,2020,An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale,A. Dosovitskiy,12,International Conference on Learning Representations,72694,"While the Transformer architecture has become the de-facto standard for natural language processing tasks, its applications to computer vision remain limited. In vision, attention is either applied in conjunction with convolutional networks, or used to replace certain components of convolutional networks while keeping their overall structure in place. We show that this reliance on CNNs is not necessary and a pure transformer applied directly to sequences of image patches can perform very well on image classification tasks. When pre-trained on large amounts of data and transferred to multiple mid-sized or small image recognition benchmarks (ImageNet, CIFAR-100, VTAB, etc.), Vision Transformer (ViT) attains excellent results compared to state-of-the-art convolutional networks while requiring substantially fewer computational resources to train." +f10a04a77fd1cd719792de374a60f3fd03f6b944,2010.09697,10.18653/v1/2021.emnlp-main.133,2020,Effects of Parameter Norm Growth During Transformer Training: Inductive Bias from Gradient Descent,William Merrill,5,Conference on Empirical Methods in Natural Language Processing,46,"The capacity of neural networks like the widely adopted transformer is known to be very high. Evidence is emerging that they learn successfully due to inductive bias in the training routine, typically a variant of gradient descent (GD). To better understand this bias, we study the tendency for transformer parameters to grow in magnitude (\ell_2 norm) during training, and its implications for the emergent representations within self attention layers. Empirically, we document norm growth in the training of transformer language models, including T5 during its pretraining. As the parameters grow in magnitude, we prove that the network approximates a discretized network with saturated activation functions. Such “saturated” networks are known to have a reduced capacity compared to the full network family that can be described in terms of formal languages and automata. Our results suggest saturation is a new characterization of an inductive bias implicit in GD of particular interest for NLP. We leverage the emergent discrete structure in a saturated transformer to analyze the role of different attention heads, finding that some focus locally on a small number of positions, while other heads compute global averages, allowing counting. We believe understanding the interplay between these two capabilities may shed further light on the structure of computation within large transformers." +e6c561d02500b2596a230b341a8eb8b921ca5bf2,2001.08361,,2020,Scaling Laws for Neural Language Models,J. Kaplan,10,arXiv.org,9468,"We study empirical scaling laws for language model performance on the cross-entropy loss. The loss scales as a power-law with model size, dataset size, and the amount of compute used for training, with some trends spanning more than seven orders of magnitude. Other architectural details such as network width or depth have minimal effects within a wide range. Simple equations govern the dependence of overfitting on model/dataset size and the dependence of training speed on model size. These relationships allow us to determine the optimal allocation of a fixed compute budget. Larger models are significantly more sample-efficient, such that optimally compute-efficient training involves training very large models on a relatively modest amount of data and stopping significantly before convergence." +3c8a456509e6c0805354bd40a35e3f2dbf8069b1,1912.01703,,2019,"PyTorch: An Imperative Style, High-Performance Deep Learning Library",A. Paszke,21,Neural Information Processing Systems,56887,"Deep learning frameworks have often focused on either usability or speed, but not both. PyTorch is a machine learning library that shows that these two goals are in fact compatible: it was designed from first principles to support an imperative and Pythonic programming style that supports code as a model, makes debugging easy and is consistent with other popular scientific computing libraries, while remaining efficient and supporting hardware accelerators such as GPUs. In this paper, we detail the principles that drove the implementation of PyTorch and how they are reflected in its architecture. We emphasize that every aspect of PyTorch is a regular Python program under the full control of its user. We also explain how the careful and pragmatic implementation of the key components of its runtime enables them to work together to achieve compelling performance. We demonstrate the efficiency of individual subsystems, as well as the overall speed of PyTorch on several commonly used benchmarks." +6c4b76232bb72897685d19b3d264c6ee3005bc2b,1910.10683,,2019,Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer,Colin A. Raffel,9,Journal of machine learning research,27969,"Transfer learning, where a model is first pre-trained on a data-rich task before being fine-tuned on a downstream task, has emerged as a powerful technique in natural language processing (NLP). The effectiveness of transfer learning has given rise to a diversity of approaches, methodology, and practice. In this paper, we explore the landscape of transfer learning techniques for NLP by introducing a unified framework that converts every language problem into a text-to-text format. Our systematic study compares pre-training objectives, architectures, unlabeled datasets, transfer approaches, and other factors on dozens of language understanding tasks. By combining the insights from our exploration with scale and our new ""Colossal Clean Crawled Corpus"", we achieve state-of-the-art results on many benchmarks covering summarization, question answering, text classification, and more. To facilitate future work on transfer learning for NLP, we release our dataset, pre-trained models, and code." +2bf7c350a8280e7c593d46a60127f99b21517121,1908.03265,,2019,On the Variance of the Adaptive Learning Rate and Beyond,Liyuan Liu,7,International Conference on Learning Representations,2325,"The learning rate warmup heuristic achieves remarkable success in stabilizing training, accelerating convergence and improving generalization for adaptive stochastic optimization algorithms like RMSprop and Adam. Here, we study its mechanism in details. Pursuing the theory behind warmup, we identify a problem of the adaptive learning rate (i.e., it has problematically large variance in the early stage), suggest warmup works as a variance reduction technique, and provide both empirical and theoretical evidence to verify our hypothesis. We further propose RAdam, a new variant of Adam, by introducing a term to rectify the variance of the adaptive learning rate. Extensive experimental results on image classification, language modeling, and neural machine translation verify our intuition and demonstrate the effectiveness and robustness of our proposed method. All implementations are available at: this https URL." +b5246fa284f86b544a7c31f050b3bd0defd053fd,1808.06226,10.18653/v1/D18-2012,2018,SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processing,Taku Kudo,2,Conference on Empirical Methods in Natural Language Processing,4390,"This paper describes SentencePiece, a language-independent subword tokenizer and detokenizer designed for Neural-based text processing, including Neural Machine Translation. It provides open-source C++ and Python implementations for subword units. While existing subword segmentation tools assume that the input is pre-tokenized into word sequences, SentencePiece can train subword models directly from raw sentences, which allows us to make a purely end-to-end and language independent system. We perform a validation experiment of NMT on English-Japanese machine translation, and find that it is possible to achieve comparable accuracy to direct subword training from raw sentences. We also compare the performance of subword training and segmentation with various configurations. SentencePiece is available under the Apache 2 license at https://github.com/google/sentencepiece." +f59c0e2a74fe6559955522e392db19e693ffbd14,,10.1007/springerreference_115624,2018,Gradient Descent,J. Gajera,2,Algorithms for Convex Optimization,102, +7a84a692327534fd227fa1e07fcb3816b633c591,1806.07572,,2018,Neural Tangent Kernel: Convergence and Generalization in Neural Networks,Arthur Jacot,3,Neural Information Processing Systems,4393,"At initialization, artificial neural networks (ANNs) are equivalent to Gaussian processes in the infinite-width limit, thus connecting them to kernel methods. We prove that the evolution of an ANN during training can also be described by a kernel: during gradient descent on the parameters of an ANN, the network function (which maps input vectors to output vectors) follows the so-called kernel gradient associated with a new object, which we call the Neural Tangent Kernel (NTK). This kernel is central to describe the generalization features of ANNs. While the NTK is random at initialization and varies during training, in the infinite-width limit it converges to an explicit limiting kernel and stays constant during training. This makes it possible to study the training of ANNs in function space instead of parameter space. Convergence of the training can then be related to the positive-definiteness of the limiting NTK. We then focus on the setting of least-squares regression and show that in the infinite-width limit, the network function follows a linear differential equation during training. The convergence is fastest along the largest kernel principal components of the input data with respect to the NTK, hence suggesting a theoretical motivation for early stopping. Finally we study the NTK numerically, observe its behavior for wide networks, and compare it to the infinite-width limit." +54a13bcc9613dcaa76fb25fbe96572f376cfcca9,1804.04235,,2018,Adafactor: Adaptive Learning Rates with Sublinear Memory Cost,Noam Shazeer,2,International Conference on Machine Learning,1400,"In several recently proposed stochastic optimization methods (e.g. RMSProp, Adam, Adadelta), parameter updates are scaled by the inverse square roots of exponential moving averages of squared past gradients. Maintaining these per-parameter second-moment estimators requires memory equal to the number of parameters. For the case of neural network weight matrices, we propose maintaining only the per-row and per-column sums of these moving averages, and estimating the per-parameter second moments based on these sums. We demonstrate empirically that this method produces similar results to the baseline. Secondly, we show that adaptive methods can produce larger-than-desired updates when the decay rate of the second moment accumulator is too slow. We propose update clipping and a gradually increasing decay rate scheme as remedies. Combining these methods and dropping momentum, we achieve comparable results to the published Adam regime in training the Transformer model on the WMT 2014 English-German machine translation task, while using very little auxiliary storage in the optimizer. Finally, we propose scaling the parameter updates based on the scale of the parameters themselves." +8691706ad0cf5e83969658b2e6bfffdc379440c9,1801.10198,,2018,Generating Wikipedia by Summarizing Long Sequences,Peter J. Liu,7,International Conference on Learning Representations,886,"We show that generating English Wikipedia articles can be approached as a multi- document summarization of source documents. We use extractive summarization to coarsely identify salient information and a neural abstractive model to generate the article. For the abstractive model, we introduce a decoder-only architecture that can scalably attend to very long sequences, much longer than typical encoder- decoder architectures used in sequence transduction. We show that this model can generate fluent, coherent multi-sentence paragraphs and even whole Wikipedia articles. When given reference documents, we show it can extract relevant factual information as reflected in perplexity, ROUGE scores and human evaluations." +d07284a6811f1b2745d91bdb06b040b57f226882,,,2017,Decoupled Weight Decay Regularization,I. Loshchilov,2,International Conference on Learning Representations,41248, +204e3073870fae3d05bcbc2f6a8e263d9b72e776,1706.03762,,2017,Attention is All you Need,Ashish Vaswani,8,Neural Information Processing Systems,199000,"The dominant sequence transduction models are based on complex recurrent or convolutional neural networks in an encoder-decoder configuration. The best performing models also connect the encoder and decoder through an attention mechanism. We propose a new simple network architecture, the Transformer, based solely on attention mechanisms, dispensing with recurrence and convolutions entirely. Experiments on two machine translation tasks show these models to be superior in quality while being more parallelizable and requiring significantly less time to train. Our model achieves 28.4 BLEU on the WMT 2014 English-to-German translation task, improving over the existing best results, including ensembles by over 2 BLEU. On the WMT 2014 English-to-French translation task, our model establishes a new single-model state-of-the-art BLEU score of 41.8 after training for 3.5 days on eight GPUs, a small fraction of the training costs of the best models from the literature. We show that the Transformer generalizes well to other tasks by applying it successfully to English constituency parsing both with large and limited training data." +2dfeb5a90abc49ab2a80a492a01a4e2c8e92ec22,1704.04760,10.1145/3079856.3080246,2017,In-datacenter performance analysis of a tensor processing unit,Norman P. Jouppi,76,International Symposium on Computer Architecture,5532,"Many architects believe that major improvements in cost-energy-performance must now come from domain-specific hardware. This paper evaluates a custom ASIC-called a Tensor Processing Unit (TPU)-deployed in datacenters since 2015 that accelerates the inference phase of neural networks (NN). The heart of the TPU is a 65,536 8-bit MAC matrix multiply unit that offers a peak throughput of 92 TeraOps/second (TOPS) and a large (28 MiB) software-managed on-chip memory. The TPU's deterministic execution model is a better match to the 99th-percentile response-time requirement of our NN applications than are the time-varying optimizations of CPUs and GPUs that help average throughput more than guaranteed latency. The lack of such features helps explain why, despite having myriad MACs and a big memory, the TPU is relatively small and low power. We compare the TPU to a server-class Intel Haswell CPU and an Nvidia K80 GPU, which are contemporaries deployed in the same datacenters. Our workload, written in the high-level TensorFlow framework, uses production NN applications (MLPs, CNNs, and LSTMs) that represent 95% of our datacenters' NN inference demand. Despite low utilization for some applications, the TPU is on average about 15X–30X faster than its contemporary GPU or CPU, with TOPS/Watt about 30X–80X higher. Moreover, using the GPU's GDDR5 memory in the TPU would triple achieved TOPS and raise TOPS/Watt to nearly 70X the GPU and 200X the CPU." +63e39cdf1ad884da6bc69096bb3413b5b1100559,1608.05859,10.18653/V1/E17-2025,2016,Using the Output Embedding to Improve Language Models,Ofir Press,2,Conference of the European Chapter of the Association for Computational Linguistics,871,"We study the topmost weight matrix of neural network language models. We show that this matrix constitutes a valid word embedding. When training language models, we recommend tying the input embedding and this output embedding. We analyze the resulting update rules and show that the tied embedding evolves in a more similar way to the output embedding than to the input embedding in the untied model. We also offer a new method of regularizing the output embedding. Our methods lead to a significant reduction in perplexity, as we are able to show on a variety of neural network language models. Finally, we show that weight tying can reduce the size of neural translation models to less than half of their original size without harming their performance." +b022f2a277a4bf5f42382e86e4380b96340b9e86,1608.03983,,2016,SGDR: Stochastic Gradient Descent with Warm Restarts,I. Loshchilov,2,International Conference on Learning Representations,11715,"Restart techniques are common in gradient-free optimization to deal with multimodal functions. Partial warm restarts are also gaining popularity in gradient-based optimization to improve the rate of convergence in accelerated gradient schemes to deal with ill-conditioned functions. In this paper, we propose a simple warm restart technique for stochastic gradient descent to improve its anytime performance when training deep neural networks. We empirically study its performance on the CIFAR-10 and CIFAR-100 datasets, where we demonstrate new state-of-the-art results at 3.14% and 16.21%, respectively. We also demonstrate its advantages on a dataset of EEG recordings and on a downsampled version of the ImageNet dataset. Our source code is available at this https URL" +97fb4e3d45bb098e27e0071448b6152217bd35a5,1607.06450,,2016,Layer Normalization,Jimmy Ba,3,arXiv.org,13427,"Training state-of-the-art, deep neural networks is computationally expensive. One way to reduce the training time is to normalize the activities of the neurons. A recently introduced technique called batch normalization uses the distribution of the summed input to a neuron over a mini-batch of training cases to compute a mean and variance which are then used to normalize the summed input to that neuron on each training case. This significantly reduces the training time in feed-forward neural networks. However, the effect of batch normalization is dependent on the mini-batch size and it is not obvious how to apply it to recurrent neural networks. In this paper, we transpose batch normalization into layer normalization by computing the mean and variance used for normalization from all of the summed inputs to the neurons in a layer on a single training case. Like batch normalization, we also give each neuron its own adaptive bias and gain which are applied after the normalization but before the non-linearity. Unlike batch normalization, layer normalization performs exactly the same computation at training and test times. It is also straightforward to apply to recurrent neural networks by computing the normalization statistics separately at each time step. Layer normalization is very effective at stabilizing the hidden state dynamics in recurrent networks. Empirically, we show that layer normalization can substantially reduce the training time compared with previously published techniques." +de5e7320729f5d3cbb6709eb6329ec41ace8c95d,1606.08415,,2016,Gaussian Error Linear Units (GELUs),Dan Hendrycks,2,,7599,"We propose the Gaussian Error Linear Unit (GELU), a high-performing neural network activation function. The GELU activation function is $x\Phi(x)$, where $\Phi(x)$ the standard Gaussian cumulative distribution function. The GELU nonlinearity weights inputs by their value, rather than gates inputs by their sign as in ReLUs ($x\mathbf{1}_{x>0}$). We perform an empirical evaluation of the GELU nonlinearity against the ReLU and ELU activations and find performance improvements across all considered computer vision, natural language processing, and speech tasks." +ea9d2a2b4ce11aaf85136840c65f3bc9c03ab649,,,2010,Understanding the difficulty of training deep feedforward neural networks,X. Glorot,2,International Conference on Artificial Intelligence and Statistics,19828, +,,,2023,"Flax: A neural network library and ecosystem for JAX, 2023",,0,URL http: //github.com/google/flax,0, +,,,2022,"Hyung Won Chung, Charles Sutton",,0,Scaling language modeling with pathways,0, +6c7384845f7d8347a6daf393ce1586e03dca971b,,,2021,Tensor Programs IV: Feature Learning in Infinite-Width Neural Networks,Greg Yang,2,International Conference on Machine Learning,315, +,,,2020,The DeepMind JAX Ecosystem,,0,http://github.com/ deepmind,0, +9405cc0d6169988371b2755e573cc28650d14dfe,,,2019,Language Models are Unsupervised Multitask Learners,A. Radford,6,,30929, +,,,2018,"JAX: composable transformations of Python+NumPy programs, 2018",,0,URL http://github.com/google/ jax,0, +2db6b8e36bb5a3594b2200b2d4f931268d6279db,,10.18653/v1/p17-2,2017,Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers),R. Barzilay,2,Annual Meeting of the Association for Computational Linguistics,220, +,,,,A random walk model of transformer parameter growth,,0,,0, +,,,,"Grain - feeding jax models, 2023",,0,http://github.com/google/grain,0, +,,,,"contributed to the infrastructure used for experimentation, provided key insight related to parameterization, and contributed to the writing",,0,,0, +,,,,Intriguing properties of transformer training instabilities,,0,To appear,0, diff --git a/02-literature/search/results/snowball-back-zhai2023.csv b/02-literature/search/results/snowball-back-zhai2023.csv new file mode 100644 index 0000000..2f5e931 --- /dev/null +++ b/02-literature/search/results/snowball-back-zhai2023.csv @@ -0,0 +1,73 @@ +s2_id,arxiv,doi,year,title,first_author,n_authors,venue,citations,abstract +37c1a63152c219ee8f630ecdea18f782074389e9,2212.06138,10.48550/arXiv.2212.06138,2022,CLIP Itself is a Strong Fine-tuner: Achieving 85.7% and 88.0% Top-1 Accuracy with ViT-B and ViT-L on ImageNet,Xiaoyi Dong,10,arXiv.org,58,"Recent studies have shown that CLIP has achieved remarkable success in performing zero-shot inference while its fine-tuning performance is not satisfactory. In this paper, we identify that fine-tuning performance is significantly impacted by hyper-parameter choices. We examine various key hyper-parameters and empirically evaluate their impact in fine-tuning CLIP for classification tasks through a comprehensive study. We find that the fine-tuning performance of CLIP is substantially underestimated. Equipped with hyper-parameter refinement, we demonstrate CLIP itself is better or at least competitive in fine-tuning compared with large-scale supervised pre-training approaches or latest works that use CLIP as prediction targets in Masked Image Modeling. Specifically, CLIP ViT-Base/16 and CLIP ViT-Large/14 can achieve 85.7%,88.0% finetuning Top-1 accuracy on the ImageNet-1K dataset . These observations challenge the conventional conclusion that CLIP is not suitable for fine-tuning, and motivate us to rethink recently proposed improvements based on CLIP. We will release our code publicly at \url{https://github.com/LightDXY/FT-CLIP}." +84f6ab620eb7b112e5b2ca64b305970894e679c1,2207.14484,10.48550/arXiv.2207.14484,2022,Adaptive Gradient Methods at the Edge of Stability,Jeremy M. Cohen,11,arXiv.org,105,"Very little is known about the training dynamics of adaptive gradient methods like Adam in deep learning. In this paper, we shed light on the behavior of these algorithms in the full-batch and sufficiently large batch settings. Specifically, we empirically demonstrate that during full-batch training, the maximum eigenvalue of the preconditioned Hessian typically equilibrates at a certain numerical value -- the stability threshold of a gradient descent algorithm. For Adam with step size $\eta$ and $\beta_1 = 0.9$, this stability threshold is $38/\eta$. Similar effects occur during minibatch training, especially as the batch size grows. Yet, even though adaptive methods train at the ``Adaptive Edge of Stability'' (AEoS), their behavior in this regime differs in a significant way from that of non-adaptive methods at the EoS. Whereas non-adaptive algorithms at the EoS are blocked from entering high-curvature regions of the loss landscape, adaptive gradient methods at the AEoS can keep advancing into high-curvature regions, while adapting the preconditioner to compensate. Our findings can serve as a foundation for the community's future understanding of adaptive gradient methods in deep learning." +5eeb80dc67590422db64ca95ec0aded24799cfb6,2206.03126,10.48550/arXiv.2206.03126,2022,Signal Propagation in Transformers: Theoretical Perspectives and the Role of Rank Collapse,Lorenzo Noci,6,Neural Information Processing Systems,171,"Transformers have achieved remarkable success in several domains, ranging from natural language processing to computer vision. Nevertheless, it has been recently shown that stacking self-attention layers - the distinctive architectural component of Transformers - can result in rank collapse of the tokens' representations at initialization. The question of if and how rank collapse affects training is still largely unanswered, and its investigation is necessary for a more comprehensive understanding of this architecture. In this work, we shed new light on the causes and the effects of this phenomenon. First, we show that rank collapse of the tokens' representations hinders training by causing the gradients of the queries and keys to vanish at initialization. Furthermore, we provide a thorough description of the origin of rank collapse and discuss how to prevent it via an appropriate depth-dependent scaling of the residual branches. Finally, our analysis unveils that specific architectural hyperparameters affect the gradients of queries and values differently, leading to disproportionate gradient norms. This suggests an explanation for the widespread use of adaptive methods for Transformers' optimization." +d2f63b56fc6bc373f5c023454c2b253326962865,2204.07118,10.48550/arXiv.2204.07118,2022,DeiT III: Revenge of the ViT,Hugo Touvron,3,European Conference on Computer Vision,729,"A Vision Transformer (ViT) is a simple neural architecture amenable to serve several computer vision tasks. It has limited built-in architectural priors, in contrast to more recent architectures that incorporate priors either about the input data or of specific tasks. Recent works show that ViTs benefit from self-supervised pre-training, in particular BerT-like pre-training like BeiT. In this paper, we revisit the supervised training of ViTs. Our procedure builds upon and simplifies a recipe introduced for training ResNet-50. It includes a new simple data-augmentation procedure with only 3 augmentations, closer to the practice in self-supervised learning. Our evaluations on Image classification (ImageNet-1k with and without pre-training on ImageNet-21k), transfer learning and semantic segmentation show that our procedure outperforms by a large margin previous fully supervised training recipes for ViT. It also reveals that the performance of our ViT trained with supervision is comparable to that of more recent architectures. Our results could serve as better baselines for recent self-supervised approaches demonstrated on ViT." +a6e231db70a6774abc995afc0201671e0dab4d3a,2204.07141,10.48550/arXiv.2204.07141,2022,Masked Siamese Networks for Label-Efficient Learning,Mahmoud Assran,9,European Conference on Computer Vision,456,"We propose Masked Siamese Networks (MSN), a self-supervised learning framework for learning image representations. Our approach matches the representation of an image view containing randomly masked patches to the representation of the original unmasked image. This self-supervised pre-training strategy is particularly scalable when applied to Vision Transformers since only the unmasked patches are processed by the network. As a result, MSNs improve the scalability of joint-embedding architectures, while producing representations of a high semantic level that perform competitively on low-shot image classification. For instance, on ImageNet-1K, with only 5,000 annotated images, our base MSN model achieves 72.4% top-1 accuracy, and with 1% of ImageNet-1K labels, we achieve 75.7% top-1 accuracy, setting a new state-of-the-art for self-supervised learning on this benchmark. Our code is publicly available." +0b0d7d87c58d41b92d907347b778032be5966f60,2203.03466,10.48550/arXiv.2203.03466,2022,Tensor Programs V: Tuning Large Neural Networks via Zero-Shot Hyperparameter Transfer,Greg Yang,10,arXiv.org,340,"Hyperparameter (HP) tuning in deep learning is an expensive process, prohibitively so for neural networks (NNs) with billions of parameters. We show that, in the recently discovered Maximal Update Parametrization (muP), many optimal HPs remain stable even as model size changes. This leads to a new HP tuning paradigm we call muTransfer: parametrize the target model in muP, tune the HP indirectly on a smaller model, and zero-shot transfer them to the full-sized model, i.e., without directly tuning the latter at all. We verify muTransfer on Transformer and ResNet. For example, 1) by transferring pretraining HPs from a model of 13M parameters, we outperform published numbers of BERT-large (350M parameters), with a total tuning cost equivalent to pretraining BERT-large once; 2) by transferring from 40M parameters, we outperform published numbers of the 6.7B GPT-3 model, with tuning cost only 7% of total pretraining cost. A Pytorch implementation of our technique can be found at github.com/microsoft/mup and installable via `pip install mup`." +e0995bad59c8638ea8c319bb7220c0f0b1ed5dca,2203.00555,10.1109/TPAMI.2024.3386927,2022,"DeepNet: Scaling Transformers to 1,000 Layers",Hongyu Wang,6,IEEE Transactions on Pattern Analysis and Machine Intelligence,284,"In this paper, we propose a simple yet effective method to stabilize extremely deep Transformers. Specifically, we introduce a new normalization function (DeepNorm) to modify the residual connection in Transformer, accompanying with theoretically derived initialization. In-depth theoretical analysis shows that model updates can be bounded in a stable way. The proposed method combines the best of two worlds, i.e., good performance of Post-LN and stable training of Pre-LN, making DeepNorm a preferred alternative. We successfully scale Transformers up to 1,000 layers (i.e., 2,500 attention and feed-forward network sublayers) without difficulty, which is one order of magnitude deeper than previous deep Transformers. Extensive experiments demonstrate that DeepNet has superior performance across various benchmarks, including machine translation, language modeling (i.e., BERT, GPT) and vision pre-training (i.e., BEiT). Remarkably, on a multilingual benchmark with 7,482 translation directions, our 200-layer model with 3.2B parameters significantly outperforms the 48-layer state-of-the-art model with 12B parameters by 5 BLEU points, which indicates a promising scaling direction." +00754af9cd0eaf613a99b13fb19e422b09beeee4,2202.00980,,2022,Robust Training of Neural Networks using Scale Invariant Architectures,Zhiyuan Li,5,International Conference on Machine Learning,37,"In contrast to SGD, adaptive gradient methods like Adam allow robust training of modern deep networks, especially large language models. However, the use of adaptivity not only comes at the cost of extra memory but also raises the fundamental question: can non-adaptive methods like SGD enjoy similar benefits? In this paper, we provide an affirmative answer to this question by proposing to achieve both robust and memory-efficient training via the following general recipe: (1) modify the architecture and make it scale invariant, i.e. the scale of parameter doesn't affect the output of the network, (2) train with SGD and weight decay, and optionally (3) clip the global gradient norm proportional to weight norm multiplied by $\sqrt{\tfrac{2\lambda}{\eta}}$, where $\eta$ is learning rate and $\lambda$ is weight decay. We show that this general approach is robust to rescaling of parameter and loss by proving that its convergence only depends logarithmically on the scale of initialization and loss, whereas the standard SGD might not even converge for many initializations. Following our recipe, we design a scale invariant version of BERT, called SIBERT, which when trained simply by vanilla SGD achieves performance comparable to BERT trained by adaptive methods like Adam on downstream tasks." +6351ebb4a3287f5f3e1273464b3b91e5df5a16d7,2111.06377,10.1109/CVPR52688.2022.01553,2021,Masked Autoencoders Are Scalable Vision Learners,Kai-Ming He,6,Computer Vision and Pattern Recognition,13366,"This paper shows that masked autoencoders (MAE) are scalable self-supervised learners for computer vision. Our MAE approach is simple: we mask random patches of the input image and reconstruct the missing pixels. It is based on two core designs. First, we develop an asymmetric encoder-decoder architecture, with an encoder that operates only on the visible subset of patches (without mask tokens), along with a lightweight decoder that reconstructs the original image from the latent representation and mask tokens. Second, we find that masking a high proportion of the input image, e.g., 75%, yields a nontrivial and meaningful self-supervisory task. Coupling these two designs enables us to train large models efficiently and effectively: we accelerate training (by 3× or more) and improve accuracy. Our scalable approach allows for learning high-capacity models that generalize well: e.g., a vanilla ViT-Huge model achieves the best accuracy (87.8%) among methods that use only ImageNet-1K data. Transfer performance in downstream tasks outperforms supervised pretraining and shows promising scaling behavior." +2582a04918f6fe62dc142f2fca9ca0bb0b1d7895,2110.09456,,2021,NormFormer: Improved Transformer Pretraining with Extra Normalization,Sam Shleifer,3,arXiv.org,102,"During pretraining, the Pre-LayerNorm transformer suffers from a gradient magnitude mismatch: gradients at early layers are much larger than at later layers. These issues can be alleviated by our proposed NormFormer architecture, which adds three normalization operations to each layer: a Layer Norm after self attention, head-wise scaling of self-attention outputs, and a Layer Norm after the first fully connected layer. The extra operations incur negligible compute cost (+0.4% parameter increase), but improve pretraining perplexity and downstream task performance for both causal and masked language models ranging from 125 Million to 2.7 Billion parameters. For example, adding NormFormer on top of our strongest 1.3B parameter baseline can reach equal perplexity 24% faster, or converge 0.27 perplexity better in the same compute budget. This model reaches GPT3-Large (1.3B) zero shot performance 60% faster. For masked language modeling, NormFormer improves fine-tuned GLUE performance by 1.9% on average. Code to train NormFormer models is available in fairseq https://github.com/pytorch/fairseq/tree/main/examples/normformer ." +83474f6510e0b985595f936d233321e131966bae,2110.04369,,2021,A Loss Curvature Perspective on Training Instability in Deep Learning,J. Gilmer,9,arXiv.org,53,"In this work, we study the evolution of the loss Hessian across many classification tasks in order to understand the effect the curvature of the loss has on the training dynamics. Whereas prior work has focused on how different learning rates affect the loss Hessian observed during training, we also analyze the effects of model initialization, architectural choices, and common training heuristics such as gradient clipping and learning rate warmup. Our results demonstrate that successful model and hyperparameter choices allow the early optimization trajectory to either avoid -- or navigate out of -- regions of high curvature and into flatter regions that tolerate a higher learning rate. Our results suggest a unifying perspective on how disparate mitigation strategies for training instability ultimately address the same underlying failure mode of neural network optimization, namely poor conditioning. Inspired by the conditioning perspective, we show that learning rate warmup can improve training stability just as much as batch normalization, layer normalization, MetaInit, GradInit, and Fixup initialization." +2a805d0e1b067444a554c5169d189fa1f649f411,2106.04560,10.1109/CVPR52688.2022.01179,2021,Scaling Vision Transformers,Xiao-Hua Zhai,4,Computer Vision and Pattern Recognition,1538,"Attention-based neural networks such as the Vision Transformer (ViT) have recently attained state-of-the-art results on many computer vision benchmarks. Scale is a primary ingredient in attaining excellent results, therefore, understanding a model's scaling properties is a key to designing future generations effectively. While the laws for scaling Transformer language models have been studied, it is unknown how Vision Transformers scale. To address this, we scale ViT models and data, both up and down, and characterize the relationships between error rate, data, and compute. Along the way, we refine the architecture and training of ViT, reducing memory consumption and increasing accuracy of the resulting models. As a result, we successfully train a ViT model with two billion parameters, which attains a new state-of-the-art on ImageNet of 90.45% top-1 accuracy. The model also performs well for few-shot transfer, for example, reaching 84.86% top-1 accuracy on ImageNet with only 10 examples per class." +7509c66a666e2e3f14bc8676b969b945ee6e136f,2106.03143,,2021,CAPE: Encoding Relative Positions with Continuous Augmented Positional Embeddings,Tatiana Likhomanenko,5,Neural Information Processing Systems,74,"Without positional information, attention-based Transformer neural networks are permutation-invariant. Absolute or relative positional embeddings are the most popular ways to feed Transformer models with positional information. Absolute positional embeddings are simple to implement, but suffer from generalization issues when evaluating on sequences longer than seen at training time. Relative positions are more robust to input length change, but are more complex to implement and yield inferior model throughput due to extra computational and memory costs. In this paper, we propose an augmentation-based approach (CAPE) for absolute positional embeddings, which keeps the advantages of both absolute (simplicity and speed) and relative positional embeddings (better generalization). In addition, our empirical evaluation on state-of-the-art models in machine translation, image and speech recognition demonstrates that CAPE leads to better generalization performance as well as increased stability with respect to training hyper-parameters." +42a7015e48a1e00b70ebb442a82afb4b10017c0b,2106.01548,,2021,When Vision Transformers Outperform ResNets without Pretraining or Strong Data Augmentations,Xiangning Chen,3,International Conference on Learning Representations,406,"Vision Transformers (ViTs) and MLPs signal further efforts on replacing hand-wired features or inductive biases with general-purpose neural architectures. Existing works empower the models by massive data, such as large-scale pre-training and/or repeated strong data augmentations, and still report optimization-related problems (e.g., sensitivity to initialization and learning rates). Hence, this paper investigates ViTs and MLP-Mixers from the lens of loss geometry, intending to improve the models' data efficiency at training and generalization at inference. Visualization and Hessian reveal extremely sharp local minima of converged models. By promoting smoothness with a recently proposed sharpness-aware optimizer, we substantially improve the accuracy and robustness of ViTs and MLP-Mixers on various tasks spanning supervised, adversarial, contrastive, and transfer learning (e.g., +5.3\% and +11.0\% top-1 accuracy on ImageNet for ViT-B/16 and Mixer-B/16, respectively, with the simple Inception-style preprocessing). We show that the improved smoothness attributes to sparser active neurons in the first few layers. The resultant ViTs outperform ResNets of similar size and throughput when trained from scratch on ImageNet without large-scale pre-training or strong data augmentations. Model checkpoints are available at \url{https://github.com/google-research/vision_transformer}." +ad4a0938c48e61b7827869e4ac3baffd0aefab35,2104.14294,10.1109/ICCV48922.2021.00951,2021,Emerging Properties in Self-Supervised Vision Transformers,Mathilde Caron,7,IEEE International Conference on Computer Vision,10753,"In this paper, we question if self-supervised learning provides new properties to Vision Transformer (ViT) [16] that stand out compared to convolutional networks (convnets). Beyond the fact that adapting self-supervised methods to this architecture works particularly well, we make the following observations: first, self-supervised ViT features contain explicit information about the semantic segmentation of an image, which does not emerge as clearly with supervised ViTs, nor with convnets. Second, these features are also excellent k-NN classifiers, reaching 78.3% top-1 on ImageNet with a small ViT. Our study also underlines the importance of momentum encoder [26], multi-crop training [9], and the use of small patches with ViTs. We implement our findings into a simple self-supervised method, called DINO, which we interpret as a form of self-distillation with no labels. We show the synergy between DINO and ViTs by achieving 80.1% top-1 on ImageNet in linear evaluation with ViT-Base." +0d5406775fab3e71848908327fb5504df5f60f92,2104.10972,,2021,ImageNet-21K Pretraining for the Masses,T. Ridnik,4,NeurIPS Datasets and Benchmarks,1011,"ImageNet-1K serves as the primary dataset for pretraining deep learning models for computer vision tasks. ImageNet-21K dataset, which is bigger and more diverse, is used less frequently for pretraining, mainly due to its complexity, low accessibility, and underestimation of its added value. This paper aims to close this gap, and make high-quality efficient pretraining on ImageNet-21K available for everyone. Via a dedicated preprocessing stage, utilization of WordNet hierarchical structure, and a novel training scheme called semantic softmax, we show that various models significantly benefit from ImageNet-21K pretraining on numerous datasets and tasks, including small mobile-oriented models. We also show that we outperform previous ImageNet-21K pretraining schemes for prominent new models like ViT and Mixer. Our proposed pretraining pipeline is efficient, accessible, and leads to SoTA reproducible results, from a publicly available dataset. The training code and pretrained models are available at: https://github.com/Alibaba-MIIL/ImageNet21K" +739ceacfafb1c4eaa17509351b647c773270b3ae,2104.02057,10.1109/ICCV48922.2021.00950,2021,An Empirical Study of Training Self-Supervised Vision Transformers,Xinlei Chen,3,IEEE International Conference on Computer Vision,2549,"This paper does not describe a novel method. Instead, it studies a straightforward, incremental, yet must-know baseline given the recent progress in computer vision: self-supervised learning for Vision Transformers (ViT). While the training recipes for standard convolutional networks have been highly mature and robust, the recipes for ViT are yet to be built, especially in the self-supervised scenarios where training becomes more challenging. In this work, we go back to basics and investigate the effects of several fundamental components for training self-supervised ViT. We observe that instability is a major issue that degrades accuracy, and it can be hidden by apparently good results. We reveal that these results are indeed partial failure, and they can be improved when training is made more stable. We benchmark ViT results in MoCo v3 and several other self-supervised frameworks, with ablations in various aspects. We discuss the currently positive evidence as well as challenges and open questions. We hope that this work will provide useful data points and experience for future research." +dfb37e6216e792bf6bd5a30c0fc7ad55df1cb71e,2103.03404,,2021,Attention is Not All You Need: Pure Attention Loses Rank Doubly Exponentially with Depth,Yihe Dong,3,International Conference on Machine Learning,643,"Attention-based architectures have become ubiquitous in machine learning, yet our understanding of the reasons for their effectiveness remains limited. This work proposes a new way to understand self-attention networks: we show that their output can be decomposed into a sum of smaller terms, each involving the operation of a sequence of attention heads across layers. Using this decomposition, we prove that self-attention possesses a strong inductive bias towards""token uniformity"". Specifically, without skip connections or multi-layer perceptrons (MLPs), the output converges doubly exponentially to a rank-1 matrix. On the other hand, skip connections and MLPs stop the output from degeneration. Our experiments verify the identified convergence phenomena on different variants of standard transformer architectures." +1480446723a1aefc26ebb0e6f105c35ab52ddb42,2103.00065,,2021,Gradient Descent on Neural Networks Typically Occurs at the Edge of Stability,Jeremy M. Cohen,5,International Conference on Learning Representations,522,"We empirically demonstrate that full-batch gradient descent on neural network training objectives typically operates in a regime we call the Edge of Stability. In this regime, the maximum eigenvalue of the training loss Hessian hovers just above the numerical value $2 / \text{(step size)}$, and the training loss behaves non-monotonically over short timescales, yet consistently decreases over long timescales. Since this behavior is inconsistent with several widespread presumptions in the field of optimization, our findings raise questions as to whether these presumptions are relevant to neural network training. We hope that our findings will inspire future efforts aimed at rigorously understanding optimization at the Edge of Stability. Code is available at https://github.com/locuslab/edge-of-stability." +2b8088253e2378fce001a090fe923b81e8dedf25,2101.03697,10.1109/CVPR46437.2021.01352,2021,RepVGG: Making VGG-style ConvNets Great Again,Xiaohan Ding,6,Computer Vision and Pattern Recognition,2619,"We present a simple but powerful architecture of convolutional neural network, which has a VGG-like inference-time body composed of nothing but a stack of 3 × 3 convolution and ReLU, while the training-time model has a multi-branch topology. Such decoupling of the training-time and inference-time architecture is realized by a structural re-parameterization technique so that the model is named RepVGG. On ImageNet, RepVGG reaches over 80% top-1 accuracy, which is the first time for a plain model, to the best of our knowledge. On NVIDIA 1080Ti GPU, RepVGG models run 83% faster than ResNet-50 or 101% faster than ResNet-101 with higher accuracy and show favorable accuracy-speed trade-off compared to the state-of-the-art models like EfficientNet and RegNet. The code and trained models are available at https://github.com/megvii-model/RepVGG." +0822f8d7e6a72a65e65f147d3a8d8fccd485da40,2012.15832,10.18653/v1/2021.acl-long.427,2021,Shortformer: Better Language Modeling using Shorter Inputs,Ofir Press,3,Annual Meeting of the Association for Computational Linguistics,104,"Increasing the input length has been a driver of progress in language modeling with transformers. We identify conditions where shorter inputs are not harmful, and achieve perplexity and efficiency improvements through two new methods that decrease input length. First, we show that initially training a model on short subsequences before moving on to longer ones both reduces overall training time and, surprisingly, substantially improves perplexity. Second, we show how to improve the efficiency of recurrence methods in transformers, which let models condition on previously processed tokens when generating sequences that exceed the maximal length the transformer can handle at once. Existing methods require computationally expensive relative position embeddings; we introduce a simple alternative of adding absolute position embeddings to queries and keys instead of to word embeddings, which efficiently produces superior results. We show that these recurrent models also benefit from short input lengths. Combining these techniques speeds up training by a factor of 1.65, reduces memory usage, and substantially improves perplexity on WikiText-103, without adding any parameters." +ad7ddcc14984caae308c397f1a589aae75d4ab71,2012.12877,,2020,Training data-efficient image transformers & distillation through attention,Hugo Touvron,6,International Conference on Machine Learning,10113,"Recently, neural networks purely based on attention were shown to address image understanding tasks such as image classification. However, these visual transformers are pre-trained with hundreds of millions of images using an expensive infrastructure, thereby limiting their adoption. In this work, we produce a competitive convolution-free transformer by training on Imagenet only. We train them on a single computer in less than 3 days. Our reference vision transformer (86M parameters) achieves top-1 accuracy of 83.1% (single-crop evaluation) on ImageNet with no external data. More importantly, we introduce a teacher-student strategy specific to transformers. It relies on a distillation token ensuring that the student learns from the teacher through attention. We show the interest of this token-based distillation, especially when using a convnet as a teacher. This leads us to report results competitive with convnets for both Imagenet (where we obtain up to 85.2% accuracy) and when transferring to other tasks. We share our code and models." +62ce4d65335c32844247e0ecf3be6de1ccb924b2,2010.11745,10.21437/interspeech.2021-1758,2020,Rethinking Evaluation in ASR: Are Our Models Robust Enough?,Tatiana Likhomanenko,8,Interspeech,117,"Is pushing numbers on a single benchmark valuable in automatic speech recognition? Research results in acoustic modeling are typically evaluated based on performance on a single dataset. While the research community has coalesced around various benchmarks, we set out to understand generalization performance in acoustic modeling across datasets -- in particular, if models trained on a single dataset transfer to other (possibly out-of-domain) datasets. Further, we demonstrate that when a large enough set of benchmarks is used, average word error rate (WER) performance over them provides a good proxy for performance on real-world data. Finally, we show that training a single acoustic model on the most widely-used datasets -- combined -- reaches competitive performance on both research and real-world benchmarks." +7f0c7c324675179f0e32c160d99c7066c7ab30ae,2010.11524,10.21437/interspeech.2021-740,2020,slimIPL: Language-Model-Free Iterative Pseudo-Labeling,Tatiana Likhomanenko,5,Interspeech,79,"Recent results in end-to-end ASR have demonstrated the efficacy of simple pseudo-labeling for semi-supervised models trained both with Connectionist Temporal Classification (CTC) and Sequence-to-Sequence (seq2seq) losses. Iterative Pseudo-Labeling (IPL), which continuously trains a single model using pseudo-labels iteratively re-generated as the model learns, has been shown to further increase performance in ASR. We improve upon the IPL algorithm: as the model learns, we propose to iteratively re-generate transcriptions with hard labels (the most probable tokens) assignments, that is without a language model. We call this approach Language-Model-Free IPL (slimIPL) and we give a resultant training setup for CTC and seq2seq models. At inference, our experiments show that decoding with a strong language model is more beneficial with slimIPL than IPL, asIPL exhibits some language model over-fitting issues. Compared to prior work on semi-supervised and unsupervised approaches, slimIPL not only simplifies the training process, but also achieves competitive and state-of-the-art results on LibriSpeech test sets in both standard and low-resource settings." +268d347e8a55b5eb82fb5e7d2f800e33c75ab18a,2010.11929,,2020,An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale,A. Dosovitskiy,12,International Conference on Learning Representations,72694,"While the Transformer architecture has become the de-facto standard for natural language processing tasks, its applications to computer vision remain limited. In vision, attention is either applied in conjunction with convolutional networks, or used to replace certain components of convolutional networks while keeping their overall structure in place. We show that this reliance on CNNs is not necessary and a pure transformer applied directly to sequences of image patches can perform very well on image classification tasks. When pre-trained on large amounts of data and transferred to multiple mid-sized or small image recognition benchmarks (ImageNet, CIFAR-100, VTAB, etc.), Vision Transformer (ViT) attains excellent results compared to state-of-the-art convolutional networks while requiring substantially fewer computational resources to train." +a2cd073b57be744533152202989228cb4122270a,2010.01412,,2020,Sharpness-Aware Minimization for Efficiently Improving Generalization,Pierre Foret,4,International Conference on Learning Representations,2174,"In today's heavily overparameterized models, the value of the training loss provides few guarantees on model generalization ability. Indeed, optimizing only the training loss value, as is commonly done, can easily lead to suboptimal model quality. Motivated by the connection between geometry of the loss landscape and generalization---including a generalization bound that we prove here---we introduce a novel, effective procedure for instead simultaneously minimizing loss value and loss sharpness. In particular, our procedure, Sharpness-Aware Minimization (SAM), seeks parameters that lie in neighborhoods having uniformly low loss; this formulation results in a min-max optimization problem on which gradient descent can be performed efficiently. We present empirical results showing that SAM improves model generalization across a variety of benchmark datasets (e.g., CIFAR-{10, 100}, ImageNet, finetuning tasks) and models, yielding novel state-of-the-art performance for several. Additionally, we find that SAM natively provides robustness to label noise on par with that provided by state-of-the-art procedures that specifically target learning with noisy labels." +b5271f4522fd72e335535c5f65d3afc01d1cb2bd,2008.07772,,2020,Very Deep Transformers for Neural Machine Translation,Xiao-Dong Liu,4,arXiv.org,111,"We explore the application of very deep Transformer models for Neural Machine Translation (NMT). Using a simple yet effective initialization technique that stabilizes training, we show that it is feasible to build standard Transformer-based models with up to 60 encoder layers and 12 decoder layers. These deep models outperform their baseline 6-layer counterparts by as much as 2.5 BLEU, and achieve new state-of-the-art benchmark results on WMT14 English-French (43.8 BLEU and 46.4 BLEU with back-translation) and WMT14 English-German (30.1 BLEU).The code and trained models will be publicly available at: this https URL." +e00484961fb2f30d2d48a5f9853fa3ebab140cac,,,2020,Improving Transformer Optimization Through Better Initialization,Xiao-Shi Huang,4,International Conference on Machine Learning,194, +8d908042f139575d6688c745e94156c9df6eae07,2004.08249,10.18653/v1/2020.emnlp-main.463,2020,Understanding the Difficulty of Training Transformers,Liyuan Liu,5,Conference on Empirical Methods in Natural Language Processing,328,"Transformers have been proved effective for many deep learning tasks. Training transformers, however, requires non-trivial efforts regarding carefully designing learning rate schedulers and cutting-edge optimizers (the standard SGD fails to train Transformers effectively). In this paper, we study Transformer training from both theoretical and empirical perspectives. Our analysis reveals that unbalanced gradients are not the root cause of the instability of training. Instead, we identify an amplification effect that substantially influences training. Specifically, we observe that for each layer in a multi-layer Transformer model, heavy dependency on its residual branch makes training unstable since it amplifies small parameter perturbations (e.g., parameter updates) and result in significant disturbances in the model output, yet a light dependency limits the potential of model training and can lead to an inferior trained model. Inspired by our analysis, we propose Admin ($\mathbf{Ad}$aptive $\mathbf{m}$odel $\mathbf{in}$itialization) to stabilize the training in the early stage and unleash its full potential in the late stage. Extensive experiments show that Admin is more stable, converges faster, and leads to better performance." +e52051204cb1179584f3b008c9d38848b52c1f28,2003.04887,,2020,ReZero is All You Need: Fast Convergence at Large Depth,Thomas C. Bachlechner,5,Conference on Uncertainty in Artificial Intelligence,462,"Deep networks often suffer from vanishing or exploding gradients due to inefficient signal propagation, leading to long training times or convergence difficulties. Various architecture designs, sophisticated residual-style networks, and initialization schemes have been shown to improve deep signal propagation. Recently, Pennington et al. used free probability theory to show that dynamical isometry plays an integral role in efficient deep learning. We show that the simplest architecture change of gating each residual connection using a single zero-initialized parameter satisfies initial dynamical isometry and outperforms more complex approaches. Although much simpler than its predecessors, this gate enables training thousands of fully connected layers with fast convergence and better test performance for ResNets trained on CIFAR-10. We apply this technique to language modeling and find that we can easily train 120-layer Transformers. When applied to 12 layer Transformers, it converges 56% faster on enwiki8." +7af72a461ed7cda180e7eab878efd5f35d79bbf4,2002.05709,,2020,A Simple Framework for Contrastive Learning of Visual Representations,Ting Chen,4,International Conference on Machine Learning,26915,"This paper presents SimCLR: a simple framework for contrastive learning of visual representations. We simplify recently proposed contrastive self-supervised learning algorithms without requiring specialized architectures or a memory bank. In order to understand what enables the contrastive prediction tasks to learn useful representations, we systematically study the major components of our framework. We show that (1) composition of data augmentations plays a critical role in defining effective predictive tasks, (2) introducing a learnable nonlinear transformation between the representation and the contrastive loss substantially improves the quality of the learned representations, and (3) contrastive learning benefits from larger batch sizes and more training steps compared to supervised learning. By combining these findings, we are able to considerably outperform previous methods for self-supervised and semi-supervised learning on ImageNet. A linear classifier trained on self-supervised representations learned by SimCLR achieves 76.5% top-1 accuracy, which is a 7% relative improvement over previous state-of-the-art, matching the performance of a supervised ResNet-50. When fine-tuned on only 1% of the labels, we achieve 85.8% top-5 accuracy, outperforming AlexNet with 100X fewer labels." +6e5d89c2b3b5ead2c3ab389534de62a28c1e8e6e,1912.07145,10.1109/BigData50022.2020.9378171,2019,PyHessian: Neural Networks Through the Lens of the Hessian,Z. Yao,4,2020 IEEE International Conference on Big Data (Big Data),423,"We present PYHESSIAN, a new scalable framework that enables fast computation of Hessian (i.e., second-order derivative) information for deep neural networks. PYHESSIAN enables fast computations of the top Hessian eigenvalues, the Hessian trace, and the full Hessian eigenvalue/spectral density; it supports distributed-memory execution on cloud/supercomputer systems; and it is available as open source [1]. This general framework can be used to analyze neural network models, including the topology of the loss landscape (i.e., curvature information) to gain insight into the behavior of different models/optimizers. As an example, we analyze the effect of residual connections and Batch Normalization layers on the trainability of neural networks. One recent claim, based on simpler first-order analysis, is that residual connections and Batch Normalization make the loss landscape ""smoother,"" thus making it easier for Stochastic Gradient Descent to converge to a good solution. Our second-order analysis, easily enabled by PYHESSIAN, shows new finer-scale insights, demonstrating that while conventional wisdom is sometimes validated, in other cases it is simply incorrect. In particular, we find that Batch Normalization does not necessarily make the loss landscape smoother, especially for shallow networks." +703685e969fed715e13937c11d7ecc5cc7c4dfd0,1910.05895,10.5281/zenodo.3525484,2019,Transformers without Tears: Improving the Normalization of Self-Attention,Toan Q. Nguyen,2,International Workshop on Spoken Language Translation,288,"We evaluate three simple, normalization-centric changes to improve Transformer training. First, we show that pre-norm residual connections (PRENORM) and smaller initializations enable warmup-free, validation-based training with large learning rates. Second, we propose l2 normalization with a single scale parameter (SCALENORM) for faster training and better performance. Finally, we reaffirm the effectiveness of normalizing word embeddings to a fixed length (FIXNORM). On five low-resource translation pairs from TED Talks-based corpora, these changes always converge, giving an average +1.1 BLEU over state-of-the-art bilingual baselines and a new 32.8 BLEU on IWSLT '15 English-Vietnamese. We ob- serve sharper performance curves, more consistent gradient norms, and a linear relationship between activation scaling and decoder depth. Surprisingly, in the high-resource setting (WMT '14 English-German), SCALENORM and FIXNORM remain competitive but PRENORM degrades performance." +b0fae9fbb4e580d92395eabafe73e317ae6510e3,1904.08779,10.21437/Interspeech.2019-2680,2019,SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognition,Daniel S. Park,7,Interspeech,4310,"We present SpecAugment, a simple data augmentation method for speech recognition. SpecAugment is applied directly to the feature inputs of a neural network (i.e., filter bank coefficients). The augmentation policy consists of warping the features, masking blocks of frequency channels, and masking blocks of time steps. We apply SpecAugment on Listen, Attend and Spell networks for end-to-end speech recognition tasks. We achieve state-of-the-art performance on the LibriSpeech 960h and Swichboard 300h tasks, outperforming all prior work. On LibriSpeech, we achieve 6.8% WER on test-other without the use of a language model, and 5.8% WER with shallow fusion with a language model. This compares to the previous state-of-the-art hybrid system of 7.5% WER. For Switchboard, we achieve 7.2%/14.6% on the Switchboard/CallHome portion of the Hub5'00 test set without the use of a language model, and 6.8%/14.1% with shallow fusion, which compares to the previous state-of-the-art hybrid system at 8.3%/17.3% WER." +faadd7d081c8d67e8c2567e8a5579e46cd6b2280,1904.01038,10.18653/v1/N19-4009,2019,"fairseq: A Fast, Extensible Toolkit for Sequence Modeling",Myle Ott,8,North American Chapter of the Association for Computational Linguistics,3444,"fairseq is an open-source sequence modeling toolkit that allows researchers and developers to train custom models for translation, summarization, language modeling, and other text generation tasks. The toolkit is based on PyTorch and supports distributed training across multiple GPUs and machines. We also support fast mixed-precision training and inference on modern GPUs. A demo video can be found at https://www.youtube.com/watch?v=OtgDdWtHvto" +409b9abbeecbc91d00187e73d441f556c8c6b598,1901.10159,,2019,An Investigation into Neural Net Optimization via Hessian Eigenvalue Density,B. Ghorbani,3,International Conference on Machine Learning,494,"To understand the dynamics of optimization in deep neural networks, we develop a tool to study the evolution of the entire Hessian spectrum throughout the optimization process. Using this, we study a number of hypotheses concerning smoothness, curvature, and sharpness in the deep learning literature. We then thoroughly analyze a crucial structural feature of the spectra: in non-batch normalized networks, we observe the rapid appearance of large isolated eigenvalues in the spectrum, along with a surprising concentration of the gradient in the corresponding eigenspaces. In batch normalized networks, these two effects are almost absent. We characterize these effects, and explain how they affect optimization speed through both theory and experiments. As part of this work, we adapt advanced tools from numerical linear algebra that allow scalable and accurate estimation of the entire Hessian spectrum of ImageNet-scale neural networks; this technique may be of independent interest in other applications." +96c82727dd5a80fef93007f888bb8569feb6bd85,1901.09321,,2019,Fixup Initialization: Residual Learning Without Normalization,Hongyi Zhang,3,International Conference on Learning Representations,430,"Normalization layers are a staple in state-of-the-art deep neural network architectures. They are widely believed to stabilize training, enable higher learning rate, accelerate convergence and improve generalization, though the reason for their effectiveness is still an active research topic. In this work, we challenge the commonly-held beliefs by showing that none of the perceived benefits is unique to normalization. Specifically, we propose fixed-update initialization (Fixup), an initialization motivated by solving the exploding and vanishing gradient problem at the beginning of training via properly rescaling a standard initialization. We find training residual networks with Fixup to be as stable as training with normalization -- even for networks with 10,000 layers. Furthermore, with proper regularization, Fixup enables residual networks without normalization to achieve state-of-the-art performance in image classification and machine translation." +d170bd486e4c0fe82601e322b0e9e0dde63ab299,1809.10853,,2018,Adaptive Input Representations for Neural Language Modeling,Alexei Baevski,2,International Conference on Learning Representations,450,"We introduce adaptive input representations for neural language modeling which extend the adaptive softmax of Grave et al. (2017) to input representations of variable capacity. There are several choices on how to factorize the input and output layers, and whether to model words, characters or sub-word units. We perform a systematic comparison of popular choices for a self-attentional architecture. Our experiments show that models equipped with adaptive embeddings are more than twice as fast to train than the popular character input CNN while having a lower number of parameters. On the WikiText-103 benchmark we achieve 18.7 perplexity, an improvement of 10.5 perplexity compared to the previously best published result and on the Billion Word benchmark, we achieve 23.02 perplexity." +c8efcc854d97dfc2a42b83316a2109f9d166e43f,1803.02155,10.18653/v1/N18-2074,2018,Self-Attention with Relative Position Representations,Peter Shaw,3,North American Chapter of the Association for Computational Linguistics,3001,"Relying entirely on an attention mechanism, the Transformer introduced by Vaswani et al. (2017) achieves state-of-the-art results for machine translation. In contrast to recurrent and convolutional neural networks, it does not explicitly model relative or absolute position information in its structure. Instead, it requires adding representations of absolute positions to its inputs. In this work we present an alternative approach, extending the self-attention mechanism to efficiently consider representations of the relative positions, or distances between sequence elements. On the WMT 2014 English-to-German and English-to-French translation tasks, this approach yields improvements of 1.3 BLEU and 0.3 BLEU over absolute position representations, respectively. Notably, we observe that combining relative and absolute position representations yields no further improvement in translation quality. We describe an efficient implementation of our method and cast it as an instance of relation-aware self-attention mechanisms that can generalize to arbitrary graph-labeled inputs." +84de7d27e2f6160f634a483e8548c499a2cda7fa,1802.05957,,2018,Spectral Normalization for Generative Adversarial Networks,Takeru Miyato,4,International Conference on Learning Representations,5267,"One of the challenges in the study of generative adversarial networks is the instability of its training. In this paper, we propose a novel weight normalization technique called spectral normalization to stabilize the training of the discriminator. Our new normalization technique is computationally light and easy to incorporate into existing implementations. We tested the efficacy of spectral normalization on CIFAR10, STL-10, and ILSVRC2012 dataset, and we experimentally confirmed that spectrally normalized GANs (SN-GANs) is capable of generating images of better or equal quality relative to the previous training stabilization techniques." +1e3d18beaf3921f561e1b999780f29f2b23f3b7d,1708.03888,,2017,Large Batch Training of Convolutional Networks,Yang You,3,,995,"A common way to speed up training of large convolutional networks is to add computational units. Training is then performed using data-parallel synchronous Stochastic Gradient Descent (SGD) with mini-batch divided between computational units. With an increase in the number of nodes, the batch size grows. But training with large batch size often results in the lower model accuracy. We argue that the current recipe for large batch training (linear learning rate scaling with warm-up) is not general enough and training may diverge. To overcome this optimization difficulties we propose a new training algorithm based on Layer-wise Adaptive Rate Scaling (LARS). Using LARS, we scaled Alexnet up to a batch size of 8K, and Resnet-50 to a batch size of 32K without loss in accuracy." +204e3073870fae3d05bcbc2f6a8e263d9b72e776,1706.03762,,2017,Attention is All you Need,Ashish Vaswani,8,Neural Information Processing Systems,199000,"The dominant sequence transduction models are based on complex recurrent or convolutional neural networks in an encoder-decoder configuration. The best performing models also connect the encoder and decoder through an attention mechanism. We propose a new simple network architecture, the Transformer, based solely on attention mechanisms, dispensing with recurrence and convolutions entirely. Experiments on two machine translation tasks show these models to be superior in quality while being more parallelizable and requiring significantly less time to train. Our model achieves 28.4 BLEU on the WMT 2014 English-to-German translation task, improving over the existing best results, including ensembles by over 2 BLEU. On the WMT 2014 English-to-French translation task, our model establishes a new single-model state-of-the-art BLEU score of 41.8 after training for 3.5 days on eight GPUs, a small fraction of the training costs of the best models from the literature. We show that the Transformer generalizes well to other tasks by applying it successfully to English constituency parsing both with large and limited training data." +efbd381493bb9636f489b965a2034d529cd56bcd,1609.07843,,2016,Pointer Sentinel Mixture Models,Stephen Merity,4,International Conference on Learning Representations,4957,Recent neural network sequence models with softmax classifiers have achieved their best language modeling performance only with very large hidden states and large vocabularies. Even then they struggle to predict rare or unseen words even if the context makes the prediction unambiguous. We introduce the pointer sentinel mixture architecture for neural sequence models which has the ability to either reproduce a word from the recent context or produce a word from a standard softmax classifier. Our pointer sentinel-LSTM model achieves state of the art language modeling performance on the Penn Treebank (70.9 perplexity) while using far fewer parameters than a standard softmax LSTM. In order to evaluate how well language models can exploit longer contexts and deal with more realistic vocabularies and larger corpora we also introduce the freely available WikiText corpus. +63e39cdf1ad884da6bc69096bb3413b5b1100559,1608.05859,10.18653/V1/E17-2025,2016,Using the Output Embedding to Improve Language Models,Ofir Press,2,Conference of the European Chapter of the Association for Computational Linguistics,871,"We study the topmost weight matrix of neural network language models. We show that this matrix constitutes a valid word embedding. When training language models, we recommend tying the input embedding and this output embedding. We analyze the resulting update rules and show that the tied embedding evolves in a more similar way to the output embedding than to the input embedding in the untied model. We also offer a new method of regularizing the output embedding. Our methods lead to a significant reduction in perplexity, as we are able to show on a variety of neural network language models. Finally, we show that weight tying can reduce the size of neural translation models to less than half of their original size without harming their performance." +97fb4e3d45bb098e27e0071448b6152217bd35a5,1607.06450,,2016,Layer Normalization,Jimmy Ba,3,arXiv.org,13427,"Training state-of-the-art, deep neural networks is computationally expensive. One way to reduce the training time is to normalize the activities of the neurons. A recently introduced technique called batch normalization uses the distribution of the summed input to a neuron over a mini-batch of training cases to compute a mean and variance which are then used to normalize the summed input to that neuron on each training case. This significantly reduces the training time in feed-forward neural networks. However, the effect of batch normalization is dependent on the mini-batch size and it is not obvious how to apply it to recurrent neural networks. In this paper, we transpose batch normalization into layer normalization by computing the mean and variance used for normalization from all of the summed inputs to the neurons in a layer on a single training case. Like batch normalization, we also give each neuron its own adaptive bias and gain which are applied after the normalization but before the non-linearity. Unlike batch normalization, layer normalization performs exactly the same computation at training and test times. It is also straightforward to apply to recurrent neural networks by computing the normalization statistics separately at each time step. Layer normalization is very effective at stabilizing the hidden state dynamics in recurrent networks. Empirically, we show that layer normalization can substantially reduce the training time compared with previously published techniques." +3d2c6941a9b4608ba52b328369a3352db2092ae0,1602.07868,,2016,Weight Normalization: A Simple Reparameterization to Accelerate Training of Deep Neural Networks,T. Salimans,2,Neural Information Processing Systems,2197,"We present weight normalization: a reparameterization of the weight vectors in a neural network that decouples the length of those weight vectors from their direction. By reparameterizing the weights in this way we improve the conditioning of the optimization problem and we speed up convergence of stochastic gradient descent. Our reparameterization is inspired by batch normalization but does not introduce any dependencies between the examples in a minibatch. This means that our method can also be applied successfully to recurrent models such as LSTMs and to noise-sensitive applications such as deep reinforcement learning or generative models, for which batch normalization is less well suited. Although our method is much simpler, it still provides much of the speed-up of full batch normalization. In addition, the computational overhead of our method is lower, permitting more optimization steps to be taken in the same amount of time. We demonstrate the usefulness of our method on applications in supervised image recognition, generative modelling, and deep reinforcement learning." +34038d9424ce602d7ac917a4e582d977725d4393,,10.1109/ICASSP.2015.7178964,2015,Librispeech: An ASR corpus based on public domain audio books,Vassil Panayotov,4,"IEEE International Conference on Acoustics, Speech, and Signal Processing",8673, +a6cb366736791bcccc5c8639de5a8f9636bf87e8,1412.6980,,2014,Adam: A Method for Stochastic Optimization,Diederik P. Kingma,2,International Conference on Learning Representations,173229,"We introduce Adam, an algorithm for first-order gradient-based optimization of stochastic objective functions, based on adaptive estimates of lower-order moments. The method is straightforward to implement, is computationally efficient, has little memory requirements, is invariant to diagonal rescaling of the gradients, and is well suited for problems that are large in terms of data and/or parameters. The method is also appropriate for non-stationary objectives and problems with very noisy and/or sparse gradients. The hyper-parameters have intuitive interpretations and typically require little tuning. Some connections to related algorithms, on which Adam was inspired, are discussed. We also analyze the theoretical convergence properties of the algorithm and provide a regret bound on the convergence rate that is comparable to the best known results under the online convex optimization framework. Empirical results demonstrate that Adam works well in practice and compares favorably to other stochastic optimization methods. Finally, we discuss AdaMax, a variant of Adam based on the infinity norm." +413c1142de9d91804d6d11c67ff3fed59c9fc279,,10.5555/1953048.2021068,2011,Adaptive Subgradient Methods for Online Learning and Stochastic Optimization,John C. Duchi,3,Journal of machine learning research,11646, +d2c733e34d48784a37d717fe43d9e93277a8c53e,,10.1109/CVPR.2009.5206848,2009,ImageNet: A large-scale hierarchical image database,Jia Deng,6,2009 IEEE Conference on Computer Vision and Pattern Recognition,77302, +a0117b72a4f68d0a134e24f674ca7fd0b42663b7,,10.1177/001258063805600103,1938,Causality,Illtyd Trethowan,1,,2213, +,,,2023,"Results for MT on WMT’17 English-German data for post-LN, with or without additional σReparam, with or without residual rescaling (‘DeepNorm",,0,A PREPRINT - MARCH,0, +,,,2022,"DeepNorm, solution (it uses post-LN and rescale residual connections depending on",,0,,0, +,,,2022,"can recover from attention entropy collapse (happens in encoder layers) and nicely converge, 100L-100L suffers from it and can diverge",,0,,0, +,,,2022,The goal of this section is to understand how varying the model depth for the well-established recipes affects the training stability,,0,,0, +,,,2022,We also observed the same trend of decreasing model performance with increasing the model depth. Attention entropy is nicely bounded across all depths similarly to ASR11,,0,,0, +,,,2021,"Our main stability experiments use 40 epochs of learning rate warmup, matching the setting of Chen et al.",,0,"For more details, see Appendix D.2",0, +,,,2021,Reduced Learning Rate Warmup,,0,,0, +,,,2021,"2021) pointed out by Vimal Thilak. All investigations, experiments and related analysis done by Dan Busbridge",,0,,0, +,,,2021,2021) the authors noted that the learning rate warmup period needed extending from its typical,,0,,0, +,,,2020,"50L-50L and 100L-100L, are unable to train and we observe the same vanishing gradients problem as reported by Wang et al",,0,,0, +,,,2020,2020a) are recent works that proposed to rescale residual connections,,0,,0, +9405cc0d6169988371b2755e573cc28650d14dfe,,,2019,Language Models are Unsupervised Multitask Learners,A. Radford,6,,30929, +,,,2019,For our iterative method we use the implementation of Lanczos from CuPy,,0,Okuta et al,0, +a59da4639436f582e483347a4833e7659fd3e598,,,2017,CuPy : A NumPy-Compatible Library for NVIDIA GPU Calculations,Ryosuke Okuta,5,,674, +2e0f0bbe0a630c36ccd42e3b1b62eeb5e8dc4541,,10.1002/ZAMM.19290090206,,Praktische Verfahren der Gleichungsauflösung .,R. Mises,2,,370, +261a056f8b21918e8616a429b2df6e1d5d33be41,,,,Connectionist Temporal Classification: Labelling Unsegmented Sequence Data with Recurrent Neural Networks,A. Graves,4,,5295, +845ee9838c1f5bf63b7db2c95ec5d27af14a4e02,,,,Connectionist Temporal Classification: Labelling Unsegmented Sequences with Recurrent Neural Networks,,0,,940, +,,,,"2022) results to probe for entropy collapse phenomenon. All later experiments, Section 4.3 and Appendix F, with deep transformers and deepnorm are done by Tatiana Likhomanenko",,0,,0, +,,,,"In vision we initialize the σReparam γ term using the first singular value, computed with the SVD at weight initialization. We then use one power iteration for all further updates",,0,,0, +,,,,Automatic Speech Recognition All speech recognition experiments are done by Tatiana Likhomanenko. Shuangfei Zhai and Jason Ramapuram advised to have also large scale results,,0,,0, +,,,,"Language Modeling Initial implementation and preliminary results on applicability of σReparam to the language modeling, Appendix G, are done by Yizhe Zhang with help from Jiatao Gu",,0,,0, diff --git a/02-literature/search/results/snowball-fwd-henry2020.csv b/02-literature/search/results/snowball-fwd-henry2020.csv new file mode 100644 index 0000000..3c29e6b --- /dev/null +++ b/02-literature/search/results/snowball-fwd-henry2020.csv @@ -0,0 +1,352 @@ +s2_id,arxiv,doi,year,title,first_author,n_authors,venue,citations,abstract +85e6c6e425210b15d3f49fe0214d9d585177597e,,10.1109/TKDE.2026.3728237,2026,Toward Practical Large-Scale Dynamic Heterogeneous Graph Embedding: Cold-Start Resilient Recommendation,Ma-Biao Long,8,IEEE Transactions on Knowledge and Data Engineering,0,"Deploying dynamic heterogeneous graph embeddings in production faces key challenges of scalability, data freshness, and cold-start. This paper introduces a practical, two-stage solution that balances deep graph representation with low-latency incremental updates. Our framework combines HetSGFormer, a scalable graph transformer for static learning, with Incremental Locally Linear Embedding (ILLE), a lightweight, CPU-based algorithm for real-time updates. HetSGFormer captures global structure with linear scalability, while ILLE provides rapid, targeted updates to incorporate new data, thus avoiding costly full retraining. This dual approach is cold-start resilient, leveraging the graph to create meaningful embeddings from sparse data. On billion-scale graphs, online A/B tests show HetSGFormer achieved up to a 6.11% lift in Advertiser Value over previous methods, while the ILLE module added another 3.22% lift and reduced the embedding refresh interval by 83.2%, substantially improving recommendation timeliness. Our work provides a validated framework for deploying dynamic graph learning in production environments." +8611d92554deaa97a6866e7d768138a277bce2ab,2610.36192,,2026,Beyond the Estimate: Wiener-Informed Decoding for Easier Training of Pixel-Space Flow Models,Zi-Qiao Wang,2,,0,"Learning velocity fields directly in pixel space is difficult and slow. We show that coarse second-order image statistics can ease this task: frozen Wiener estimates improve velocity learning, with gains extending beyond the linear predictions they explicitly provide. Moreover, we observe similar but complementary gains from frozen bases and pixel decoding. These observations motivate Wiener-informed Decoding (WiD), which uses statistical knowledge both for direct prediction and to organize decoder computation without modifying native blocks. WiD combines a trainable Wiener prediction path with coefficient interfaces that inject features into the native decoder and return its predicted coefficient corrections, with these components initialized with data statistics. All introduced parameters, including the spectrum and coordinate transforms, are trained jointly under the original pixel-space velocity objective. Across model scales and decoder architectures, WiD accelerates training and improves generation quality with modest parameter overhead. It also yields smoother gradient trajectories and remains effective with representation alignment. WiD improves class-conditional and text-to-image generation across different resolutions. On ImageNet 256 x 256, PixelDiT-XL with WiD reaches FID 1.53 after 500 epochs, achieving comparable quality to PixelDiT with 37.5% fewer training epochs. Statistical denoisers contribute beyond the estimate: their computational structure can also ease pixel-space training." +8c761ed5ed749e18b4d716f5900933078680a89e,2610.34582,,2026,HobbyLM: A Sparse Mixture-of-Experts Model Family on a Hobby Budget,H. Ganesan,3,,0,"We describe HobbyLM, a family of small sparse Mixture-of-Experts (MoE) models trained from scratch with a few hundred to a few thousand dollars of rented GPU time. The core is a decoder-only transformer with fine-grained experts, sigmoid routing with auxiliary-loss-free load balancing, grouped-query attention with per-head QK-normalization, and aMuon/AdamW optimizer split. Each of these choices was selected with a single-variable ablation suite at 130M parameters. On this architecture we train a 500M-parameter model (169M active) on 40B FineWeb tokens and post-train it into a chat model, a masked-diffusion language model, a function-calling and computer-use agent, and a single""omni""checkpoint that reads text, images, video, sound and speech through frozen encoders and small projectors. We run these models on a laptop CPU with a from-scratch Rust inference engine. Two of the authors then adopted the architecture without architectural changes to build HobbyLM-1B (1.04B total, 305M active parameters), trained on a 100B-token curated mixture with a high-quality anneal. HobbyLM-1B reaches a 47.5 average on seven zero-shot benchmarks, and its builders also release an instruction-tuned model and an on-device macOS computer-use adapter. Finally, we describe HobbyLM-Image, a 333M-parameter flow-matching diffusion transformer that generates 1024x1024 images in a 32x-compressed latent space, together with a dependency-light CPU implementation of the full text-to-image pipeline. We report what worked, what did not, and where the small-scale ceilings lie. We also state for each component which author designed or built it." +413c9e6fbd32e763eed8893b7f781ef1019f7689,2610.33360,,2026,LSTM-Style Looped Transformers with Shared KV Caches,Andreas Kirsch,1,,0,"Recurrent transformers reuse weights to perform deep computation with fewer parameters. We study an LSTM-style transformer that also shares one key--value cache across its recurrent core. At width 1024, it uses 40.4% fewer parameters than an untied control at nearly matched forward FLOPs, and reduces persistent cache sets from fifteen to four. Parallel training supplies cache entries from a standard-attention pass, whereas continuation reads entries produced recursively using shared attention. We address this mismatch by aligning the two training passes'cache entries with a consistency loss during the final 10% of a fixed training-token budget. Across three model widths on FineWeb, this removes 81--89% of the continuation-loss penalty relative to full-cache evaluation. The improvement replicates on disjoint documents for two training seeds. Across four widths and 1.05--4.19B training tokens, the loss cost of weight tying is smallest at lower token budgets, suggesting an opportunity to use recurrent computation when data and model storage are limited." +7fa9e3213666de8208958ec077252402f86e98e1,2610.31633,,2026,BeNice! Closing Looping's Inference-Memory Gap via KV-Sharing,Hayden Prairie,4,,0,"Looped architectures scale quality while keeping parameter count fixed by repeatedly passing activations through the same block of layers. During inference, each recurrence generates independent KV-caches, causing memory to grow linearly. To address this challenge, prior work has proposed KV-sharing (i.e., sharing KV-caches across depth), finding that, in some cases, the KV-cache of the final recurrence can be shared with the earlier recurrences without quality loss, a property we denote as KV-shareability. However, the mechanisms underlying KV-shareability remain poorly understood, with even identically trained looped architectures not consistently exhibiting it. In this paper, we study how KV-shareability can be enabled in looped models. We propose a two-stage method: in the first stage, we use stochastic depth during pretraining, which induces contractive residual dynamics that drive representations toward a common limit across recurrences. In the second stage, we propose BeNice!, a post-training method that broadcasts the final recurrence's KV-cache into the first recurrence, training it to produce compatible residuals while preserving the original model's quality. As a result, BeNice! enables KV-shareability while matching standard finetuning's unshared validation perplexity. Using BeNice!, we show that KV-sharing is complementary to alternative KV-compression methods, composing with memory-efficient sequence mixers (e.g., multi-latent attention, Kimi delta attention) and low-precision KV-cache quantization (e.g., fp8, mxfp8, nvfp4). Finally, in simulated low-memory, long-context inference, KV-sharing improves cache hit rates by up to 31.4%, increases throughput by up to 23.6%, and reduces p90 time-to-first-token by up to 83.6%." +1a11bd7feec83fc01c829c45259533fb2eac36f1,2610.30949,,2026,IQuest-Q1: Advancing Foundation Capabilities for Agentic CLI Systems,Jian Yang,176,,0,"We present IQuest-Q1, a sparse Mixture-of-Experts (MoE) agentic foundation model with 320B total parameters (15B activated). IQuest-Q1 is developed through a three-stage process: (1) stable and efficient pre-training, combining Stable SplitNorm, Muon, and muP with a two-stage curriculum whose second stage increases the coding-data mixture, preserving a wide stability margin between safe and optimal learning rates; (2) fine-grained mid-training, expanding the data mixture across domains and formats and progressively extending the context; and (3) multi-scaffold with multi-version post-training, combining SFT, domain-specialized RL, and MOPD across various scaffolds to integrate complementary training signals and strengthen agentic coding capabilities. Furthermore, model merging combines different experts and checkpoints from different stages while mitigating conflicts among their parameter updates. The resulting IQuest-Q1 demonstrates capabilities in both coding and general tasks. We release IQuest-Q1 as an open-source model to support community research and development." +681b2df4b2c2fbf2f26e9fa5e7ec53f232ec052a,2610.31889,,2026,Scaling Transformer Energy-based Models For High Quality Image Synthesis,Jungsoo Lee,3,,0,"Energy-Based Models (EBMs) are a flexible class of generative models that decouple probabilistic modeling from the generation process. This decoupling directly enables compositional generation and inference-time scaling, where sample quality can be refined simply by increasing sampling steps. However, severe challenges in high-dimensional sampling and training instabilities with deep architectures have restricted EBMs to low-dimensional regimes. To resolve these bottlenecks, we introduce the Transformer Energy-Based Model (TEM), a scalable transformer energy function operating entirely within a latent space. On top of this backbone, we extract energies at multiple layers and sum them, composing complementary constraints from local texture to global semantics. Furthermore, two components make this high-quality at scale: Positive Representation Alignment (PRA), which supervises TEM's intermediate representations with a vision foundation model, and advanced sampling strategies that improve sample quality at inference. TEM-XL achieves an FID-50K of 3.82 on class-conditional ImageNet 256$\times$256 without classifier-free guidance, showing that a contrastive EBM can generate high-fidelity images. On ImageNet 512$\times$512, TEM achieves an FID-50K of 8.24, demonstrating that the approach scales to higher resolution. We further validate the core promises of EBMs: compositional generation, zero-shot inpainting, and out-of-distribution detection." +14032f5c5ab4caecc5fe90bc5acf48facca45150,2610.28489,,2026,Asymmetric Looped Transformers: Fixed Global Representations for an Iterative Decoder,Cheng Liang,9,,0,"Looped Transformers increase effective depth by repeatedly applying shared layers, offering a way to scale computation without increasing parameter count. However, full-stack looping places context construction and iterative state refinement within the same recurrent computation, without explicitly distinguishing their roles. We introduce the Asymmetric Looped Transformer (AsymLoop), which separates single-pass context encoding from recurrent state refinement. A single-pass causal encoder produces layer-specific Global key-value (KV) caches that remain fixed across iterations, while a weight-tied recurrent decoder generates Local KV from its evolving state. Joint attention over both sources under a single softmax allows each update to integrate persistent contextual information with the current intermediate state. For training, we apply a heavy-tail-guided layerwise learning-rate method to the encoder and decoder. Experiments on FineWeb-Edu across four model scales, from 60M to 1B, show consistently lower validation perplexity and higher average downstream accuracy than Full Loop under matched effective depth and training token budgets. At 1B, AsymLoop improves average zero-shot accuracy across seven commonsense reasoning benchmarks from 49.58% to 50.85%. Further experiments extend these benefits to longer training budgets, an OLMo-style backbone, and sparse mixture-of-experts models. These results support asymmetric context encoding and state refinement as an effective design for recurrent language models." +cda1244ee8a28649acf88c32de74af0e7aaa05f2,2610.27387,,2026,Training Controllable World Models via Action Influence Maximization,Samuel M. Bateman,8,,1,"Action-conditioned video generative models trained on successful, goal-directed demonstrations tend to ignore the commanded action, rendering a plausible expert-like future regardless of what is asked of them. We trace this failure to training data that lack sufficient action influence. We analyze this metric and prove that it upper bounds the action influence that any model trained on that data can express. Expert demonstrations, whose narrow action distributions let next states be predicted without attending to the action, set a low ceiling that no amount of model capacity alone can raise. Raising it requires collecting different data. We introduce Action Influence Maximization (AIM), an algorithm for actively collecting training data for improving dataset action influence. We compare the performance and data-efficiency of AIM against exploration heuristics and intrinsic-motivation methods on simulated navigation and manipulation tasks. In pixel-based manipulation, AIM is the only method that consistently improves controllability: 500 actively collected episodes added to 3k demonstrations match a world model trained on 300k expert episodes on end-effector controllability, an $85\times$ reduction in data." +bcba462f8b88e55de5ab1f2d5cc1e9dd6b2c793f,2610.22330,,2026,Designing Transformers for Continuous Diffusion Language Models,Wei-Jian Luo,5,,0,"Continuous diffusion language models (CDLMs) offer an alternative approach to parallel language generation; however, most existing studies directly use transformer architectures from either causal large language models (LLMs) or Diffusion Transformers (DiTs) for image generation, which were not optimized for CDLMs. In this paper, we conduct a systematic study on Transformer architectures for CDLMs and introduce Language Diffusion Transformer (LangDiT), combining feature and query-key normalization, wider noise conditioning, likelihood-shortcut gradient routing and calibration, and confidence-gated training self-conditioning. LangDiT significantly reduces generation perplexity in matched LangFlow comparisons, establishing a new state of the art in continuous language generation across different model formulations. Using LangFlow as a representative CDLM, LangDiT as a plug-and-play backbone leads to a new state-of-the-art model with a generation perplexity (GenPPL) of 21.72 using 1024 NFEs and 44.72 GenPPL using 128 NFEs on the unconditional generation of 1024-token sequences on the OpenWebText dataset. When adapted to Plaid-style CDLM formulations, LangDiT results in a new leading GenPPL of 11.33 using 1024 denoising NFEs, and 27.51 GenPPL using 128 denoising NFEs. These gains are achieved while retaining each model's underlying representation, training objective, and inference sampler, isolating transformer architecture as a fundamental source of improvement. Our study identifies architectural design as an important source of improvement in continuous diffusion language models." +53b241fec3f7fc9f50652460dcb1680ad2f9b60c,2610.21391,,2026,NOCE-Net: Representation Learning for Battery Operational Context via Nested Sequence Modelling,Arnab Bhattacharjee,3,,0,"Learning effective representations from battery operational history is essential for predictive health management and control-oriented applications. Existing approaches rely on response-dependent engineered features or short early-life windows, limiting generalisability and precluding use in prospective settings where future responses are unavailable. We propose NOCE-Net, a deep nested sequence model that learns control-enabling representations directly from raw cyclic operational or telemetric data. Our data pipeline, re-BatteryData, standardises heterogeneous aging datasets into semantically equivalent Full Equivalent Cycles (FECs) defined by absolute charge throughput, enabling end-to-end learning across diverse operational regimes. NOCE-Net combines multi-patch transformer encoders for intra-cycle dynamics with a GRU backbone for inter-cycle state evolution, operating on approximately $700$ batteries across ten public datasets spanning over $300$ operational profiles. In zero-shot evaluation on unseen battery chemistries and usage patterns, NOCE-Net achieves $13.7\%$ MAPE in voltage response prediction, outperforming strong long-sequence baselines by over $43\%$ while using $40 \times$ fewer parameters. Shape and component diagnostics of the learned hidden states reveal that the GRU's principal direction of evolution achieves Spearman $|\rho| \geq 0.95$ with the capacity fade trajectory on average across all datasets with monotone degradation profiles, including zero-shot test batteries, providing evidence that the representations encode degradation dynamics transferable to downstream tasks beyond battery response modelling. Code is available at https://github.com/BeArnab96/NOCE-Net." +cbc88335a98c1a74b6a1738f8506cdcb0320b785,2610.17994,,2026,Improving Neural Symbolic Regression with a Diverse Prior and Post-training Alignment,Dave Otte,3,,0,"Symbolic regression is the problem of finding an expression that fits a dataset. Solving it requires searching a combinatorial space of candidate expressions. Neural symbolic regression can amortize that search by training a model once on synthetic data and then predicting expressions for a new dataset. But existing models still underperform genetic programming on real, noisy data. We argue that the deficit lies in the training rather than in the paradigm, and present RASR (Reinforced Amortized Symbolic Regression), a method with two key changes to the training recipe. First, we design a synthetic prior that narrows the gap with real-world data: inputs come from the prior of a tabular foundation model; constants are sampled bottom-up, conditioned on the data; targets carry heteroscedastic and structured noise; and everything is standardized. Second, we split training into two stages. Supervised pre-training maximizes the token-level likelihood of a reference expression. This is followed by reinforcement-learning post-training that aligns the model with its inference-time use by rewarding held-out fit and penalizing expression size. On the black-box track of SRBench 2.0, RASR achieves the best mean test $R^2$ rank of all evaluated methods. Three RASR variants, trained with different size penalties, make up most of the accuracy-complexity Pareto front. On Strogatz, our method recovers more ground-truth expressions than any baseline, and on Feynman it is competitive with the best-performing methods." +6b4ef3210d16a284d304000fcd00daccbc945fed,2610.17588,,2026,Taming Softmax at the Limits of Muon: A Unified Approach through Hilbert Geometry,Aditya Somasundaram,5,,0,"Muon controls weight updates through the RMS movement of a linear layer's output. This implicitly treats the linear output as a proxy for the full module, including the nonlinearity that follows. This proxy can be misleading for softmax, where the RMS shift in activations does not faithfully measure the change in the output distribution. We propose Hilbert distance as a natural metric for bounding changes in these output distributions. Hilbert distance bounds changes in cross-entropy loss and limits probability redistribution. For softmax, it takes the form of the variation seminorm of the preactivation perturbation, allowing probability constraints to be expressed as parameter constraints. Formulating the steepest descent problem with a Hilbert distance constraint brings two recently proposed methods under a unified mathematical framework. For the LM head, solving a row-wise approximation to the resulting diameter-constrained problem leads us to RowNorm \citep{pethick2025trainingdeeplearningmodels}. For scaled dot-product attention, the same principle yields the first-order query--key steepest descent problem underlying Compositional Muon \citep{keigwin2026compositional}. These connections motivate using Hilbert distance as a guiding principle for optimizer design in softmax modules." +f8c2441bb214d2690ad6a59e4425fc0a52fc7f8b,2610.18652,,2026,AMemOps: Evaluating Memory Updates in Long Audio Streams,Pei-Wei Chang,5,,0,"Audio memory must recognise recurring sources across acoustic changes while keeping their attributes up to date. Overall accuracy does not reveal how these demands interact. We introduce Audio Memory Operations (AMemOps), a benchmark for latest-attribute retrieval, preservation of unchanged attributes and absent-source rejection in long audio streams. Paired controls vary recordings, training identities and earlier events to examine their effects on updating. Training on other recordings of the test instruments increases a reference delta-rule memory's update advantage over matched attention by 13.4 percentage points. At retrieval delays of 120\,s, matched histories separate benefits of consistent repetition from costs of conflicting earlier values, while additive-memory controls identify benefits of erasure." +e9e34af10fb91e0232f67ab79cf1f90c25dca55b,2610.18711,,2026,Steertuning Policies from a Few Human Videos,Billy Yan,11,,0,"Current recipes for robot learning require massive amounts of robot demonstrations to obtain robust and adaptive policies. In this work, we introduce Steertuning, a system that enables robots to learn new tasks in new environments from 5 human demonstrations in under 5 Steertuning consists of two parts. (1) We pretrain a 400M-parameter diffusion policy on our egocentric human dataset, which consists of 100 hours of densely annotated random interactions. This pretraining makes Steertuning robust to background differences and enables spatial and object generalization when deployed on new tasks. (2) Given 5 demonstrations of a new task, we finetune our model for only 2 minutes (2000 training updates) and steer the policy's actions during reverse sampling using the demonstration whose observation is closest to that of the deployment environment. Together with a carefully designed data processing pipeline for embodiment transfer, this enables Steertuning to achieve a 71% success rate across 12 tasks in 12 new environments, ranging from lab settings to kitchens in new homes. The entire process, from data collection for a new task to robot policy deployment, takes less than 5 minutes. Our full dataset, model weights, codebase, and robot videos will be made available at our website: https://steertuning.github.io." +0e580a9c4b85a3f7d376cdc60952ed600704ffa1,2610.16656,,2026,RFWeave: Learning Wireless Propagation Across Scenes and Transceiver Configurations,Shu-Rui Xu,8,,0,"Wireless propagation modeling is foundational to the evaluation and design of modern communication systems. To achieve accurate wireless propagation prediction, recent learning-based approaches leverage scene geometry and materials, demonstrating promising generalization across transceiver locations and unseen scenes. However, these approaches remain largely confined to a fixed configuration of array size, receiver orientation, and transmit beam. In this work, we introduce RFWeave, a learned wireless propagation model designed to generalize across scenes, transceiver locations, and transceiver configurations. To accommodate varying array structures, RFWeave incorporates variable-length transceiver tokens that represent antenna elements individually. Furthermore, to capture the geometry between transmit and receive antenna elements, RFWeave aggregates their pairwise spatial relations into propagation tokens. A configuration-conditioned decoder predicts the spatial spectrum and received power. RFWeave is trained on a large-scale simulated dataset spanning 10,000 indoor scenes and more than 5 million channel records under diverse transceiver configurations. Extensive evaluations show that RFWeave achieves state-of-the-art performance across all evaluated generalization settings, improving spatial spectrum structural similarity (SSIM) by up to 15.7% and reducing received power mean absolute error (MAE) by up to 35.4% over the strongest baselines on unseen scenes. Additional experiments on measured DICHASUS data and in beam management further demonstrate RFWeave's broader applicability to real-world channels and downstream wireless tasks." +0827c905f1b9314c23c47b1df7720e042150336f,2610.17014,,2026,MAGNET: Manifold-Aware Graph Diffusion Network for Connectome Generation,Protyay Dey,5,,0,"Functional brain connectome represents neural connectivity as a matrix of pairwise interactions between brain regions. Generation of functional connectomes is not only a question of validity; rather, having satisfied the constraints on the correlation matrix, the next step is to recover the class-conditional geometry buried under coarse labels. We propose MAGNET, a Manifold-Aware Graph Diffusion Network which uses a normalized-Cholesky representation of the manifold of correlation matrices that guarantees validity. MAGNET lifts noisy latent states into ROI-level region tokens and performs denoising with a relational inductive bias over brain atlas regions. To deal with structural problems induced by coarse labels, MAGNET employs class-anchored conditioning, amortized structural bridge, and relevance-preserving corruption. Across ABIDE, ADNI, and OASIS-3, MAGNET consistently achieves favorable results compared to previous manifold-aware approaches, demonstrating improvements of 7-21% in class-conditional fidelity ($\alpha,\beta$-F1) across three cohorts and better sampling efficiency. Moreover, while training only with strict binary labels, MAGNET is capable of maintaining clinical heterogeneity through fine substructure of the connectomes in a zero-shot setting, improving subclass covariance alignment ($\lambda$-MSE) by over 30%. These results suggest that geometric validity is a necessary but insufficient condition for clinical utility in connectome synthesis. Moreover, efforts in making the diffusion denoising class-conditional manifold aware finds utility beyond the highly curved brain connectome generation as this is a critical problem in various general settings. The code repository can be found at https://github.com/protyayofficial/MAGNET." +db00aa9feb05dcd089951f0e81aed39d69eaf0fb,2610.17202,,2026,MLAfication: Efficiently Enabling High-Fidelity MLA for Modern LLMs,Zi-Le Wang,3,,0,"Multi-Head Latent Attention (MLA) compresses the KV cache into a latent representation, reducing inference memory and bandwidth. Existing MHA2MLA/TransMLA methods do not support QKNorm and require costly recovery on modern LLMs, especially MoE. We present MLAfication, a QKNorm-compatible, parameter-efficient extension of MHA2MLA. It combines calibrated KNorm, parameter-efficient distillation, contribution-aware Partial-RoPE, and activation-aware joint KV initialization. Across MHA and GQA models, including QKNorm- and MoE-equipped variants, MLAfication preserves performance under substantial KV-cache compression, surpasses the current SOTA on commonsense and long-context benchmarks, and extends MLA conversion to modern architectures. On Qwen3-30B-A3B, MLAfication delivers up to $5.35\times$ the serving throughput of BF16 GQA and, combined with 4-bit KV quantization, reduces the KV-cache footprint by $94.73\%$. Our code is available at https://github.com/JT-Ushio/MLAfication." +d7b775107468fb1689edf2c292e4bd56b9c8a5e4,2610.17383,,2026,Variable Input Masked Autoencoders for Wrist-Worn Accelerometry,H. Haresamudram,5,,0,"Large-scale repositories of wearable sensor data have enabled the development of foundation models for human activities and behavior. However, existing models are pretrained using fixed input durations and sampling rates, despite substantial heterogeneity in recording conditions, downstream tasks, and deployment constraints. We introduce VarMAE, a variable input masked autoencoder that learns a single representation across sampling rates and temporal contexts. During pretraining we construct two views of the same signal: a long duration high frequency view, and a short duration lower frequency view, obtained via random cropping and downsampling. We perform masked reconstruction within each view and introduce a cross-resolution masked reconstruction objective that uses visible tokens from the long view to predict masked tokens from the short view. Cropping and downsampling impose complementary information bottlenecks, encouraging the model to learn representations that remain effective across input durations and sampling rate changes. We pretrain VarMAE on the large-scale NHANES dataset, producing a single pretrained encoder matching or outperforming controlled foundation model baselines on eight benchmark datasets. With only 3.82M parameters, VarMAE is parameter efficient and shows that a single foundation model can achieve broad transfer capabilities across sensing configurations." +4ba37204ea2b8f1fa61f8531f22638e5089bd719,2610.13764,,2026,How Well Do Prior-Data Fitted Networks Amortize Bayesian Inference?,M. Hulskamp,5,,0,"Prior-data fitted networks (PFNs) are trained to approximate the Bayesian posterior predictive distribution (PPD) of a synthetic prior, yet results about the approximation quality are surprisingly thin. Taking an empirical risk minimization perspective, we decompose the deployment error of PFNs into prior, context, and amortization error, and measure the latter against a reference PPD under a realistic prior. Results show that PFNs generally recover the targeted PPD very well. Nevertheless, we find that parts of the residual amortization error persist, especially in \textit{rare} settings under the prior pretraining distribution, even when scaling the model size, which manifests in disagreement between independently pretrained PFNs and leaves uncertainty about the predictor at inference time. As a remedy, we study multiple types of Bayesian model averages and find that these can consistently reduce this uncertainty and improve predictions up to full-size TabICLv2 models." +54534b46342b4d13f34fd503cea18737a5dd4cf4,2610.11376,,2026,CRISP: Fixing Flying Pixels in Latent LiDAR Generation via Diffusion Decoding,Andrea Ceron,5,,0,"Latent LiDAR pipelines suffer from flying pixels: convolutional VAEs blur sharp radial depth discontinuities, yielding edge depths that back-project to points floating between surfaces. We identify this as a major, directly correctable decoder bottleneck and introduce CRISP: a pixel-space diffusion decoder with a backbone-agnostic latent adapter, DiT-based denoiser, and support mask predictor. CRISP replaces video-VAE and LiDAR-native decoders alike while keeping the encoder and latent generator fixed. Across KITTI-360, SemanticKITTI, and nuScenes, replacing only the decoder reduces FSVD/FPVD by 50.5% on average across frozen backbones; for generic video VAEs, the reductions reach 71%/74%. On the LiDAR-native LiDM backbone, FRID drops by 71%, with the largest gains at depth discontinuities. In a pretrained LiDM world model, the same zero-shot replacement improves FSVD by 15.5%, narrowing the sim-to-real gap. Code and checkpoints will be released at https://andrea25512.github.io/CRISP/." +2bf552cca9eac56f5f498efb6f5469aa330724b8,2610.12338,,2026,VFold: Symmetry-Aware Cross-Layer Value Cache Compression,Neha Verma,4,,0,"While caching key-value (KV) states accelerates Large Language Model (LLM) decoding, this cache can dominate memory usage at long context lengths. One solution is to compress this memory by exploiting inter-layer cache similarities. However, most existing techniques necessitate architectural changes to LLMs and incur substantial overhead. In this work, we propose a symmetry-aware value cache merging strategy that reduces cache memory while avoiding both harmful performance degradation and architectural overhead during decoding. Furthermore, we show that this approach can be exploited alongside existing cache compression techniques, composing with high-ratio quantization or key cache pruning to reach compression ratios that neither method reaches alone, with minimal additional cost. Ultimately, our findings reveal a major source of underutilized capacity in the value cache, offering a simple yet highly effective direction for scaling context windows under memory constraints." +ae066d2799131136a9e49e065d211fa800cc9c4b,2610.10497,,2026,QuadTok: Quadtree Visual Tokenizer for Autoregressive Image Generation,Yu-Cheng Mao,7,,1,"We introduce QuadTok, a novel framework for visual tokenization and autoregressive image generation. Compared to traditional approaches using 2D grids or 1D token sequences, we propose a hierarchical quadtree structure, bridging the gap between 2D spatial binding and 1D sequence-level flexibility. The QuadTok tokenizer dynamically allocates representational capacity to visually intricate areas while leaving homogeneous regions at a coarse resolution. Compared with a fixed 256-token grid, our ImageNet-trained tokenizer saves approximately 10% of tokens on ImageNet and 9% when transferred zero-shot to the COCO dataset, while maintaining comparable reconstruction fidelity. Furthermore, the natural causality introduced by the tree structure seamlessly enables autoregressive image generation. Conditioned on a quadtree topology supplied before generation, our 947M GPT-style generative model achieves a 2.08 gFID on the ImageNet $256 \times 256$ benchmark. Additionally, leveraging the strong spatial correlation preserved by the quadtree structure, the QuadTok generator enables zero-shot spatially controlled image generation capabilities. Code: https://github.com/myc634/QuadTok." +73be128360856cf860401daa323151e09e95fae2,2610.09958,,2026,Force without transmission: a depth-induced rank collapse that no loss on the representation reopens,Martin Hofmann,2,,0,"Training can drive a transformer into a rank collapse: all token representations point in one direction, and learning stops. In a related collapse of attention, a loss term with a bounded corrective force repairs the network during the run. We ask whether such a term repairs rank collapse. We collapse small transformers by weakening their skip connection and treat copies of the collapsed network. No added loss term repaired the collapse, although the stronger kind pushed with about a tenth of the task gradient. The reason was the path, not the strength. The task gradient no longer reached the query and key weights, which decide where attention looks, and the added term's gradient faded before the blocks where the collapse forms. Restoring the skip connection, which changes no weight, reopened this path at once. The rank then recovered, but only far above the scale of collapse. After a burst of high learning rate the path stayed open and the rank recovered untreated. Registered predictions from the path ranked recovery times but did not transfer to this cause. In every case the loss stayed above that of a healthy network after the rank recovered. Whether a collapsed network can be repaired depends on whether the gradient still reaches the weights that must change, not on how strongly a loss term pushes." +dce8a3715c57f5a51c7bf2db2ae7c6691b9917ec,2610.07676,,2026,Exact-Solution Volume and Length Generalization in Transformers,Y. Zhu,2,,0,"Research on transformer expressivity shows whether a transformer is capable of solving a given task, but gives little indication of whether the solution, if learned, is generalizable to longer input lengths. We study this question through normalized exact-solution volume (NESV): the fraction of a bounded parameter region that achieves an exact solution on every input of length $n$. For fixed-width, single-layer transformers with $\log n$-scaled attention, we establish asymptotic bounds on NESV for four tasks: FIRST ($\Theta(1)$), MAJORITY ($\Theta(1/(n\log n))$), INDEX ($\Theta(1/n^3)$), and PARITY ($0$). These results are consistent with previous empirical results: the faster the exact-solution volume decays with input length, the harder it is to length-generalize on that task. Looking deeper into INDEX, our volume analysis reveals two error sources that grow with $n$. Consequently, we study a transformer model that would structurally eliminate one of the terms, theoretically improving the NESV bound to $\Theta(n^{-1})$, and empirically achieving 85% accuracy when tested at $10\times$ the training length, compared with the 60% accuracy of the original model. We conclude that volume analysis may be a useful approach to identify concrete sources of length sensitivity and thus provide insights into task-specific model refinements." +f0b306652620d6dd68e3b409987717c80052913e,2610.05711,,2026,"From Pixels, Without Pre-training: Joint Generative and Self-Supervised Representation Learning in One Model",Vicente Balmaseda,3,,0,"Strong image generation models are conditioned on class labels, aligned to frozen pretrained encoders, or built on separately trained autoencoders. While effective, generation then depends on supervision or pretraining: labels must be annotated, and encoders or autoencoders pretrained for the target domain. We study joint generative and self-supervised representation learning in a single model, enabling self-conditioned generation without labels or pretrained models. This is challenging because the objectives are mismatched: contrastive learning consumes clean augmented views and favors coarse, invariant semantics, while flow matching consumes noisy images and must preserve the fine detail and spatial layout that contrastive learning discards. We propose SCION (Self-conditioned Generation on Self-supervised representation), whose core is a single pixel-space encoder conditioned on the flow timestep and an embedding. For representation learning, this conditioning embedding is a learned global vector shared across images, with the encoder's [CLS] token yielding the semantic representation trained by the contrastive loss. For generative training, the conditioning embedding is the image's own [CLS] representation, while patch tokens pass through a decoder to predict the image. To sample without a reference image at inference, we jointly learn a prior over the embedding. Gradient-norm balancing and stop-gradient mechanisms enable joint optimization in one run. SCION is self-supervised and self-contained, with no labels or pretrained models. On ImageNet 256x256, with the JiT-B recipe and no representation guidance, SCION reaches 8.92 FID, surpassing class-unconditional iREPA, which aligns to pretrained DINOv2 (46.44), and RCG, which conditions on it (14.27). With JiT-L, SCION achieves 5.89 FID without guidance and 3.47 with representation guidance, outperforming RCG with the ADM recipe (6.24)." +c5ea4def5404960748affab8d54b9060fad5cd30,,10.64898/2026.10.02.756297,2026,"PIE: Generalizing perturbation effects across unseen perturbations, contexts and datasets",Rishi Verma,7,bioRxiv,1,"Predicting cellular response to perturbations is key to understanding biological mechanisms and selecting therapeutic targets. However, generalizing across cellular contexts, perturbations, and experimental datasets remains challenging due to incomplete representations of the system being perturbed and the difficulty of measuring true perturbation effects within that system. We introduce PIE, a model that addresses these limitations by reformulating the learning task around population-level perturbation effects and incorporating auxiliary inputs that better characterize the biological system. PIE directly predicts differentially expressed genes and changes in gene expression for each (context, perturbation) pair, using external biological knowledge, baseline gene expression, and observed perturbation responses. Its architecture accommodates input sources with variable feature sets and predicts effects on genes seen and unseen during training. On the Replogle–Nadig dataset, PIE achieves state-of-the-art performance on most metrics across all tested generalization settings, including the most challenging setting where both the context and perturbation are unseen during training. For predicting differentially expressed genes, PIE achieves 1.2–3.2 times the AUPRC of the strongest baseline in each setting: across unseen contexts, perturbations, experimental datasets, and jointly unseen contexts and perturbations. Overall, PIE provides a framework for generalizing perturbation effects across diverse experimental conditions." +509a0a6b0a1949ab6cec2327b0f96511f60ddc54,2610.05411,,2026,CleanMDM: Clean Motion Diffusion Model for Multimodal Motion Cleanup,Zhe Li,4,,0,"Motion capture data is rarely directly usable, as they typically exhibit missing segments, jitter, drift and contact artifacts. Traditionally, corrupted motions are cleaned by animators through the manual identification of keyframes from noisy motion, subsequent keyframe correction, and interpolation between corrected keyframes to reconstruct coherent motion. While the rise of generative motion models has made automatic cleanup feasible, most approaches operate as black box denoisers with limited controllability, making it difficult to preserve reliable segments or enforce specific user intents. Inspired by animation workflows, we present CleanMDM, a unified multimodal motion cleanup framework that formulates cleanup as masked conditional generation with plug-and-play conditions. This single model supports arbitrary combinations of noisy 3D motion, sparse 2D keyframes, sparse 3D keyframes, and text. This design enables both automatic cleanup without additional user annotation and controllable cleanup under multimodal guidance. To further improve motion realism, we incorporate the Latent Motion Quality Discriminator (LMQD) to better match kinematic distributions and reduce skating, jitter, and interpenetration artifacts, and we apply Mesh-Aware Contact Projection as a test-time optimization step to enhance contact and physical consistency. Experiments across multiple datasets demonstrate that CleanMDM consistently outperforms prior cleanup and generation baselines, and that low cost conditions (text and 2D keyframes) provide reliable controllability gains in multimodal cleanup scenarios." +2d42f51bee07f5ffd8117566975a398d7233b280,2610.03419,,2026,Deep Bayesian REFoCUS,S. Penninga,2,,1,"In this work we formulate ultrasound multistatic recovery from arbitrary transmit sequences as a Bayesian inference problem. To that end, we train a deep generative prior on multistatic data sets to tackle the rank-deficient regime in which classical linear REFoCUS decoders fail. This appproach, which we term Deep Bayesian REFoCUS, outperforms the linear baselines for all regimes of rank-deficiency and noise levels, and regresses to linear decoding when inversion is exact. The model also expresses uncertainty in the null space of the acquisitions, whereas the linear REFoCUS decoders only provide point estimates. Finally, we analyze the impact of distribution shift between simulation and in-vivo acquisitions, showing remarkable generalization ability without any fine-tuning or adaptation." +1e763b33688567fcfece68d13772878bba8ea37e,2610.01942,,2026,Latent-Foresight: End-to-End Learning Predictable Representations for Latent World Models,Efstathios Karypidis,3,,0,"Predicting the future evolution of a scene is a fundamental capability for world modeling. Recent work has shown that operating in the feature space of Vision Foundation Models (VFMs) yields semantically rich representations that support diverse future scene understanding tasks. However, existing approaches rely on two-stage pipelines, where VFM features are first compressed using fixed dimensionality reduction (e.g., PCA) or independently trained autoencoders, and a separate predictor is trained on top of the resulting frozen latent space. This decoupling between representation learning and temporal prediction, as well as approaches that apply predictors directly on raw VFM features, provides no guarantee that the latent space is structured for predictable dynamics. In this work, we propose Latent-Foresight, an end-to-end framework that jointly learns a latent tokenizer and a flow-based generative dynamics model, explicitly shaping the representation to support temporal predictability. To enable stable joint optimization, we introduce several key design choices that prevent latent collapse and align reconstruction with generative objectives. Extensive experiments show that our approach learns more temporally coherent latent representations and consistently outperforms two-stage baselines across multiple future scene understanding tasks and prediction horizons, while eliminating separate training stages, including during high-resolution adaptation. We provide the implementation code and model weights at https://github.com/Sta8is/Latent-Foresight" +78013ec3f04b3af7407246578598fd8ad87b2fe5,2609.39114,,2026,The Row Normalization Puzzle in Muon,Jia-Yu Zhang,2,,0,"This paper examines how row-wise renormalization affects Muon, focusing on the gap between NorMuon's worst-case guarantees and its practical performance (Li et al.). Despite its growing adoption and promising performance in large language model (LLM) pretraining, NorMuon's worst-case guarantees remain poorly understood. One fundamental question is: Does row normalization yield provable convergence gains, potentially through its interaction with approximate polar computation and exponential moving-average momentum? Our results show that row normalization introduces a dimension-dependent factor in the worst-case iteration complexity under the operator-norm geometry, which persists even with exact polar computation and any fixed momentum parameters. Indeed, we establish an algorithm-dependent lower bound and a matching upper bound in deterministic settings, and extend our upper bound analysis to stochastic settings. Both upper-bound analyses allow approximate polar computation. Experiments show that NorMuon is slower than Muon on synthetic problems inspired by our worst-case construction, yet outperforms Muon in LLM pretraining. These findings sharpen the puzzle of why row normalization helps in practice and complement the recent findings of Dewulf et al." +bee254c11b97a27757e60373a03fb8d4384a7a8a,2610.00487,,2026,ScaffoldM3C: A Multimodal Sequential Monte Carlo Framework for Generative Stable Construction Planning,Gadiel Sznaier Camps,5,,0,"Autonomously constructing physically realizable 3D structures remains a significant challenge due to combinatorial action spaces, interchangeable components, equifinal assembly sequences, and strict stability requirements during construction. State-of-the-art methods fine-tune large language models for text-based generative construction. However, these approaches do not allow for Multimodal (text, image, sketch) conditioning, overlook the practical role of scaffolding for stabilizing intermediate structures, and suffer from slow inference speeds. Therefore, we formulate construction as a probabilistic next-block generation task with multiple potential assembly actions and multiple potential task conditioning modalities. Concurrently, we explicitly consider the utility of scaffolding by introducing an auxiliary scaffold block token. We present Scaffold Multimodal Monte Carlo (ScaffoldM3C), a multimodal, lightweight, auto-regressive model for stable block-based construction, that proposes a set of next-step candidate blocks. Leveraging these candidates, we utilize Sequential Monte Carlo (SMC) to maintain a population of possible assembly sequences, allowing us to consider multiple, potentially different, assembly directions simultaneously. We train our multimodal architecture by extending the StableText2Brick dataset to contain image conditioning prompts and scaffold-stabilized build sequences. ScaffoldM3C is 4x smaller than competing baselines, yielding a 5x to 20x speedup during inference, while achieving comparable construction quality to state-of-the-art methods and higher overall stability. We demonstrate the effectiveness of our approach through simulations and real-world robot assembly demonstrations." +737f951651ba687fe5d8270081dfd565055edb57,2609.37038,,2026,NowcastDiT: Diffusion Transformers are Effective Precipitation Nowcasters,Hao-Ran Xu,6,,0,"Precipitation nowcasting demands accurate short-term forecasts under strong spatiotemporal variability. Diffusion models are well suited to modeling complex precipitation distributions, yet existing approaches often introduce increasingly specialized designs, leaving the capability of a standard diffusion architecture underexplored. We show that a standard Diffusion Transformer already provides a simple and scalable foundation for precipitation nowcasting, with domain-specific requirements accommodated naturally within its design space. Based on this principle, we develop NowcastDiT and instantiate this flexibility through two complementary adaptations: a dynamics-aware noise prior for temporally coherent forecasts, and end-to-end reinforcement learning with timestep-aware rewards for meteorological skill. Experiments on SEVIR and MRMS benchmarks show that NowcastDiT achieves state-of-the-art performance in both perceptual quality and meteorological skill. These results suggest that standard DiT can serve as an effective foundation for precipitation nowcasting." +e27c284e652785b05bb7c9b208e82d2ebf737057,2609.37183,,2026,HELIX: Purified and Unified - Rethinking Feature Interaction and Sequence Modeling for Large-Scale Recommendation,Yun-Tao Zheng,21,,0,"Industrial recommendation ranking models typically scale along two modeling axes: feature interaction over heterogeneous user, item, context, and cross features, and sequence modeling over long, informative, and multi-type user behavior histories. We find that scaling either capability in isolation is insufficient, as each exhibits a limited scaling ceiling and a suboptimal scaling-law slope. We conjecture that achieving a more favorable scaling-law slope requires jointly scaling both axes. To support this, we present HELIX, a purified and unified architecture for large-scale recommendation. HELIX interleaves sequence retrieval and feature interaction while enforcing one-way information flow from reusable sequence states to candidate-conditioned mix-tokens. This design preserves cross-depth communication between the two modeling axes while keeping user-side sequence computation amortizable, enabling flexible and asymmetric scaling of sequence modeling and feature interaction. Deployed in TikTok's e-commerce recommendation system, HELIX consistently improves offline CTR AUC, CVR AUC, and other ranking metrics. In online A/B tests, it achieves an approximately 6% increase in e-commerce video GMV per user." +cca9d4e0abb8f1f56226b263cd073060b2e6dc72,2609.34167,,2026,Natural Image Autoencoder-Based fMRI Representations for Trait and State Prediction,Juhyeon Park,8,,0,"Foundation models pre-trained on large-scale fMRI datasets have shown strong downstream performance, but at substantial data and computation cost. To investigate how much fMRI-specific pre-training is actually needed for such performance, we introduce FReD, which derives fMRI representations from a frozen Deep Compression AutoEncoder (DCAE) pre-trained exclusively on natural images and pairs them with a task specific readout. For trait prediction, FReD summarizes frame-wise representations by their temporal mean and log-standard deviation and applies linear probing, with late fusion across two normalization schemes. For state prediction, it represents each frame as a single token and models temporal dependencies with a shallow Transformer. Across four resting-state datasets spanning six trait-prediction targets, linear probes on frozen DCAE features generally outperform those on fMRI foundation model representations and remain competitive with fully fine-tuned fMRI foundation models. On three task-fMRI state-prediction tasks, a temporal readout on DCAE features performs comparably to the strongest foundation models evaluated. A Gaussian injection analysis further shows that localized signal changes are recovered more accurately from the frozen DCAE features than from the evaluated foundation-model representations. Together, these results show that strong performance on current fMRI benchmarks is possible without fMRI-specific representation pre-training, making frozen natural-image features as a useful baseline for assessing its added value." +b166abba59f18102e843c2326af480bfbaffb7fb,2609.34273,,2026,Query Expansion and Key Specialization in Transformer Attention Geometry,Vidit Gupta,5,,0,"The projection of queries and keys are central to the attention mechanism in Transformer architectures. While they are mathematically symmetric, they play different roles in attention mechanisms. The question of whether there is an effect from their functional distinction on their geometric development in training remains unanswered. We investigate the problem through the training of small GPT-like Transformers on character-level WikiText-103 for three different depths (4, 6, and 8 layers), three types of initialization for queries and keys, and four random seeds, resulting in 36 runs and 54 trajectories of average layers across seeds. We track the effective dimensionality of those layers using participation ratios and discover that effective dimension of queries expand while keys shrink, and that $PR_Q - PR_K$ is positive in all trajectories studied. In connection to attention, the shrinking of keys leads to a narrower spectrum of $QK^\top$ and more peaked attention weights. In order to determine if this connection is causal or coincidental, we directly control the spectrum of keys during training across five seeds: restricting it to make it shrink sharpens the attention with high directional confidence, while keeping it constant to the level of initial dispersion makes attention softer. Additional token-level checkpoint analyses show that the monotonic paired-contrast trend is not universal across pretrained families, but survives as an early-training regime that later decays over a full pretraining run, and the link between interaction-rank geometry and attention entropy remains visible in several models." +7be54510c483ee2f1389f41f7427ca5579e1ff11,2609.34272,,2026,Broken Symmetry in BF16 Attention: Why FlashAttention Gradients Blow Up Late in Training,Jun-Lin Chen,12,,0,"BF16 is now standard in large-scale pretraining, including in fused attention kernels such as FlashAttention, and these kernels are widely trusted. When we used FlashAttention-3 to pretrain a 450M-parameter transformer on 50B tokens, however, we ran into a problem: training was healthy for 25B tokens, then the gradient norm grew a thousandfold and the loss ended 0.2 nats above FP32 attention, without a single NaN. Recomputing the attention backward of just two layers in FP32 removes almost all of the excess gradient. Part of the cause is known: a fused multiply-add in the forward softmax, so far treated as an extreme-input NaN case and never fixed in FlashAttention-3. Repairing it stops the blow-up, but the query gradient is still wrong by more than its own size, and training still drives attention logits to thousands of times their size under accurate gradients. The remaining error comes from a broken conservation law. The softmax score gradient sums to zero along every row, which makes the query gradient blind to where the keys sit as a group; rounding it to BF16 leaves a small nonzero sum that leaks the mean key into the gradient, and the leak grows exactly as late training makes keys large and attention sharp. We introduce GProj (gauge projection), which restores the zero sum after the cast with two rank-one corrections per row. It cuts the remaining median query/key gradient errors from 219%/13% to 0.34%/0.37%, on par with FP32 attention, for 4.7% more time per training step. In matched from-scratch runs it trains to the same loss as FP32 attention, while FlashAttention-3 and key smoothing both destabilize." +abb7cf87a2a88295461774eb05ffa4445a8958a3,2609.34507,,2026,KiT: A Foundation Model for Financial Time-Series Forecasting using DiffusionTransformers,Bo-Yuan Zhang,2,,0,"Financial candlestick forecasting is fundamental to quantitative investment, yet it remains exceptionally challenging due to extremely low signal-to-noise ratios and vast heterogeneity across markets and instruments. Existing approaches have largely attempted to introduce deep learning to capture hidden temporal features, but most adopt an auto-regressive formulation, which leads to error accumulation during inference. Meanwhile, general-purpose time-series foundation models are not tailored to the unique structure of k-line data and yield unsatisfactory performance on downstream candlestick forecasting tasks. To tackle these problems, we introduce KiT, a K-line Diffusion Transformer foundation model, and reformulate future prediction as conditional path generation via flow matching: given a historical context window, the model generates an ensemble of plausible future OHLCV trajectories. We pre-train KiT at multiple parameter scales on billions of candlestick bars spanning multiple markets and timescales. Across three markets and seven resolutions, KiT attains a mean return RankIC of 0.057 and a mean volatility RankIC of 0.66, leading at every timescale and outperforming both task-specific financial forecasters and general time-series foundation models. Code will be available at: https://github.com/Luciferbobo/KiT." +03a48f4cbb8b566f4e5c119d06f97cfadf3282e5,2609.36227,,2026,One-Step Next-Latent Prediction Is Not a World Model,Shi-Tong Wang,3,,0,"Next-latent prediction fits a map from the current embedding to the next one. LeNEPA carries this objective to time series, replacing the stop-gradient of next-embedding prediction with the isotropy penalty of LeJEPA. A world model is a transition kernel that can be rolled out. The one-step regression identifies a conditional mean, and a mean is a kernel only in special cases. For a linear-Gaussian Markov latent, the mean transition and the innovation covariance are fixed by the one-step problem, and the open-loop squared error at horizon $K$ equals the trace of the sum of the pushed-forward innovation covariances. That error grows with $K$ after the one-step fit is exact. If the conditional mean is nonlinear, composing it is not the multi-step conditional mean. If the observation is a non-injective function of a Markov state, a memoryless one-step map does not determine future observations, while a short window can. An isotropy penalty is a function of the embedding marginal, so its partial derivative in the transition weights is zero. On a scalar autoregression with coefficient $0.9$, the one-step mean squared error is $0.998$ and the $16$-step open-loop error is $5.10$. On a hidden rotation, an eight-step window reaches $16$-step error $0.056$, while the current scalar alone reaches $0.778$. Raising the isotropy weight from $0.1$ to $10$ leaves eight-step latent error inside $[0.78,0.85]$ on three seeds." +9efd4498482121f95349681d5e93526bf037c466,2609.31525,,2026,TinyAudio: Compact and Efficient Text-to-Audio Generation for Low-Resource Deployment,Jun-Xi Liu,8,,0,"Text-to-audio (TTA) generation has advanced rapidly in generation quality and instruction following. However, representative systems often require around a billion parameters, limiting deployment on resource-constrained devices. This paper introduces TinyAudio, a compact flow-matching-based TTA model for low-resource deployment. At its core, TinyAudio uses TA-DiT, a 35M single-stream flow-matching Transformer. TinyAudio also includes TA-CLAP, a 32M audio-aligned text encoder, and TA-VAE, whose 20M decoder reconstructs 44.1 kHz audio from compressed latents. TinyAudio has only 87M parameters in total, over 90% fewer than representative billion-parameter pipelines, and uses 0.48 GB peak GPU memory. TinyAudio achieves competitive generation quality on AudioCaps and TTA-Bench. We further introduce TinyAudio-MF, a MeanFlow-accelerated model that enables real-time generation with a four-core CPU quota. Our results demonstrate a practical quality-footprint trade-off for low-resource TTA deployment." +8d159f166f843897550d568224394a4de118a843,2609.28589,,2026,"OneTrans-V2: Unifying Retrieval, Pre-rank, and Fine-rank with One Transformer in Industrial Recommender",Han-Nan Cao,21,,1,"Industrial recommendation systems typically operate as a \emph{cascade} of retrieval, pre-rank, and fine-rank, but these stages are usually trained and served as separate models, causing repeated user-sequence encoding, isolated optimization, and duplicated engineering effort. Building on OneTrans'model-level unification, we present OneTrans-V2, one Transformer that unifies the entire cascade. It encodes the user behavior sequence once as a shared context while preserving stage-specific candidate features and computation. Joint training lets the three stages reinforce one another and enables in-model knowledge distillation from fine-rank to pre-rank. We scale the shared backbone with sparse mixture-of-experts (MoE), which increases capacity with bounded activated computation, and stabilize scaling with $\mu$P-style parameterization. To consolidate objective-specific retrieval channels, we introduce Decision-Conditioned Generative Retrieval (DCGR). DCGR predicts a decision prefix describing the upcoming interaction and generates items conditioned on it, allowing business objectives to steer a single generative process. Finally, Sequence-Native Training (SNT) organizes training around each user's lifelong behavior sequence and amortizes its encoding across exposures. Deployed across all three stages of a large-scale industrial recommendation system, OneTrans-V2 improves gross merchandise value (GMV) by 9.74\% and, with a co-designed serving stack, delivers $3.2\times$ the throughput of the cascade it replaces under the same hardware budget." +6243b210e96f9204694fe8ac125d7153406924a3,2609.24559,,2026,$t_0$: A Time-Series Foundation Model for Forecasting with Context,Lucas Meyer,9,,6,"We present $t_0$, a family of open-weights foundation models for forecasting with multivariate context. We release its first two members: $\texttt{t0-alpha}$ and $\texttt{t0-beta}$, respectively 102M and 256M parameters. Both condition their forecasts on target history, past covariates, and known-future covariates, without task-specific retraining. Their transformer layers alternate attention along time and across variates. They produce probabilistic forecasts through quantile predictions. Pretraining combines curated public data with synthetic generator families constructed to contain covariate-to-target dependencies. On GIFT-Eval, $\texttt{t0-alpha}$ reaches an aggregate CRPS of 0.4941, and $\texttt{t0-beta}$ a CRPS of 0.4738 and a MASE of 0.6865, third on both and within 4.0% of the best zero-shot TSFM. On fev-bench they score 42.2 and 46.7 in skill, the latter third again and 2.0 points behind the leader. We analyze $\texttt{t0-alpha}$ in depth. Known-future covariates raise its skill by 6.3 percentage points across 30 tasks. The report also examines its calibration, its rollout strategy on long horizons, and its robustness to missing data. On the Victoria electricity-demand benchmark, $\texttt{t0-beta}$ is among the most accurate models with a context of nearly a year. In an independent Macrocosm evaluation of hourly ERCOT prices over 29 months, both cut the MAE of the lagged-price baseline by 38%." +365c0d78761e79e7638048889f2e7432f73c50f4,2609.25081,,2026,ufakzeka-1: Building and Evaluating a 151M-Parameter Turkish Language Model from Scratch,Sait Furkan Teke,1,,2,"We describe ufakzeka-1, a 151M-parameter (182M with embeddings) decoder-only Turkish language model pretrained from scratch on 13.5B tokens of openly licensed text and instruction-tuned for chat, at a total cost of about \$286 in cloud GPU, API and notebook time. The contribution is not the model's capability, which is what a model this size can be expected to have, but the record of building and measuring it: a Turkish byte-level tokenizer at 1.77 tokens per word, a three-stage pretraining schedule, a post-training mixture of openly licensed and generated data, and an evaluation battery of release gates, a rule-checked sweep of 5,508 conversations, judged conversations and hand tests, all with prompts held out from the training data, enforced by decontamination inside the data build and by a checked-in invariant script we run before each build. We report three findings that we believe transfer to other small-model efforts: a safety gate that had been""fixed""with training data written from its own questions read 64/64 while the honest figure was 34/64; training-seed variance was as large as the spread across every recipe we tried, so single-seed comparisons at this scale are uninformative; and data rounds repaired only what was absent from the data, while identity tracking over long context and multi-turn arithmetic did not move across any data change we tried, which we read as limits of the model size rather than gaps in the data, a reading the next, larger model will test. Weights, the data recipe, the evaluation code and the spend ledger are released under Apache-2.0." +45168d8a6ccbcaaeb4ead00f4c35b4a4f7bbc2cc,2609.19363,,2026,Directions That Don't Drift: Stiefel Manifold Routing for Transformer Attention,Rubén Darío Guerrero,1,,0,"The query and key projections $\WQ,\WK$ in attention are almost always trained by Euclidean optimizers with no geometric constraint. We constrain them to the Stiefel manifold and optimize with a Riemannian Adam carrying one scalar second moment per frame---the form of \citet{becigneul2019}, here extended to the compact, non-Hadamard $\St(d,r)$ with a tangent projector, step-norm cap, and polar retraction. Four propositions prove steepest descent in the embedded metric, gradient-scale independence, well-conditioning, and exact $\mathrm{O}(d)$-equivariance. A fifth records that weight decay has \emph{identically zero} Riemannian gradient on $\St(d,r)$ ($W{=}WI_r$ lies in the normal space), so decay cannot act on the constrained frames. On a CIFAR-10 patch benchmark at $n{=}10\mathrm{k}$ this rule gains $\mathbf{+6.79}$\,pp over AdamW across 12 paired starts ($t{=}38.33$, $12/12$); earlier fixed-step Riemannian SGD gains $+1.97$\,pp, of which $+1.69$\,pp comes from frozen orthonormal initialization alone. The corrected Adam's lead grows with data: $+1.9$\,pp at $n{=}1\mathrm{k}$ to $+6.7$\,pp at $n{=}50\mathrm{k}$. A 12-seed ablation credits all gain to the scale-free step ($+4.63$\,pp, $12/12$), nothing to the projector or equivariance; a targeted $\varepsilon$-sweep causally confirms the mechanism ($-2.6$\,pp at $\varepsilon{=}0.1$, $p{<}0.001$). Two five-seed grokking studies confirm the constrained arm does not grok better than the baseline ($p{=}0.019$, A2 wins): the weight-decay exemption has no grokking consequence. A single-seed pilot exploiting this localization achieves the first stable grokking under slingshot conditions---Stiefel + targeted circuit regularization keeps routing-frame isometry error $10^6\times$ lower than the unconstrained ablation through every collapse." +aa9f64b800689fe7c24371bd98d3cbc578eda84f,2609.22322,10.64898/2026.09.18.752552,2026,Spike Sorting with VanillaSort,Zi-Shuo Feng,2,bioRxiv,0,"Training spike detectors on real recordings is challenging because algorithmically generated labels can be noisy and incomplete. We propose VanillaSort, combining multichannel detection with spatially augmented, template-guided clustering. VanillaDet uses visibility-aware masking, truncated Gaussian targets and a temporally tolerant positive-bag loss, followed by conditional event-SNR gating. VanillaCluster combines HuiduRep embeddings with relative-amplitude features for Gaussian mixture clustering and refines assignments using cross-fitted waveform templates built from selected core events. VanillaDet improves detection accuracy over SimSort by two and three percentage points on the static and drift subsets of Hybrid Janelia, respectively. The complete pipeline also improves sorting performance over the corresponding HuiduRep baselines. These results support learning from imperfect real-data labels and incorporating waveform consistency into neuronal assignment." +72ecdb770e845d192e4d618c06c43600bd522aab,2609.15032,,2026,Tele360: Real-Time Feed-Forward Human Reconstruction from Sparse Unposed Cameras,Han-Zhang Tu,5,,0,"Live free-viewpoint visualization of real humans is critical for immersive communication and interactive digital experiences. Existing methods either rely on computationally expensive optimization or require calibrated cameras and low-resolution inputs, making real-time high-resolution deployment impractical. In this work, we present Tele360, the first real-time feed-forward system for dynamic human reconstruction and live free-viewpoint visualization from sparse, unposed RGB streams. Our system jointly estimates camera poses and reconstructs a dynamic 3D Gaussian representation for each time instance in a single forward pass. To achieve this, we start by designing a lightweight sparsity-aware multi-view transformer backbone that tokenizes foreground human regions while preserving global context through a shared scene token. We then employ a fully transformer-based Gaussian decoder to mitigate convolution-induced over-smoothing while keeping decoding sparse and efficient. In addition, we introduce a hybrid feature pyramid that injects multi-scale appearance cues into geometry prediction. We further introduce a lightweight differentiable Levenberg-Marquardt camera refinement layer to enhance multi-view consistency and geometric alignment. Moreover, to stabilize learning under sparse, unposed inputs, we transfer multi-view geometry priors from a large visual-geometry foundation model via teacher-student distillation. Finally, the predicted Gaussian maps are streamed with video codecs to remote devices for interactive free-viewpoint rendering. Extensive experiments show that Tele360 achieves state-of-the-art visual quality on studio benchmarks while supporting real-time 2K input-to-rendering at over 25 FPS on a single consumer GPU. Additional captured sequences illustrate its performance across varied subjects, clothing, and motions under our multi-camera setup." +953a3a4bf9b50e2eab46deb45c7aae1ee41cb597,2609.14715,,2026,Depth and Scale in the Sub-150M Regime: JugnuLM-53M vs JugnuLM-110M,D. Rajput,3,,0,"We scale our conventional sub-150M pretraining recipe from 53.5M to 109.7M parameters, holding the method fixed (Qwen3-style decoder with grouped-query attention, RoPE, SwiGLU, RMSNorm, QK-Norm, and a z-loss; FineWeb-Edu data) and changing only the geometry to a deep-and-thin 23-layer x 576-hidden design. The larger model improves across the board -- BLiMP 78.1 ->81.3, ARC-Easy 51.4 ->52.5, WikiText-2 byte-perplexity 2.04 ->1.95 -- and its 81.3% BLiMP essentially matches GPT-X2-125M (81.28) at about 12% fewer parameters. Notably the 110M model achieves this on fewer training tokens (about 8B vs 12B), so the gain is attributable to capacity and depth, not more data. Both models are deliberately conventional; this report is a clean scaling control and the baseline rung (R0) of an ablation study of what further improves models in this regime. An ablation ladder follows: value residuals (R1) and the Muon optimizer (R2) lift ARC-Easy by a cumulative +3.6 (52.5 ->56.1) at a near-flat BLiMP and are kept; a diverse data blend (R3) and two logit-distillation settings (R4a/R4b) are not kept -- honest negatives. R3 pins ARC-Easy to FineWeb-Edu's educational filtering rather than raw diversity; distillation from a 1.7B teacher can reach the class-leading ARC-Easy (56.99, matching GPT-X2-125M) but only at a perplexity cost that dialing KD down then erases -- so R2 remains the best kept stack." +b096d1fcb4b1b05f4dc8fa20a437d4920ade4949,2609.09157,,2026,Learning Length-Extrapolatable Recurrent Models,Hanwen Jiang,1,,0,"Recurrent models provide a natural path to long-context modeling, yet models trained with backpropagation through time (BPTT) often fail beyond their training horizon. Classical analyses emphasize gradients that vanish or explode along temporal paths. However, dense per-token losses can still train a shared recurrent rule despite severe decay, showing that decay alone does not determine whether learning fails. We instead study state credit: the signal through which future losses reach earlier recurrent states before contributing to parameter updates. Accordingly, we intervene directly on state credit and propose Credit Stabilization through Time (CST). During backward propagation, CST locally rescales the state-credit signal to stabilize its norm without rotating the component being corrected, while leaving the forward computation unchanged. Because controlled synthetic tasks and real data exhibit different credit dynamics, we specialize CST to each regime. In both settings, CST improves performance beyond the training horizon, with gains observed at up to 128x the training length." +5a6481bd7917dee6062a927f2283899e6901e329,2609.07086,,2026,Conditioned Initialization for Attention,Hemanth Saratchandran,2,,0,"Transformers are a dominant architecture in modern machine learning, powering applications across vision, language, and beyond. At the core of their success lies the attention layer, where the query, key, and value matrices determine how token dependencies are captured. While considerable work has focused on scaling and optimizing Transformers, comparatively little attention has been paid to how the weights of the queries, keys and values are initialized. Common practice relies on random initialization or alternatives such as mimetic initialization, which imitates weight patterns from converged models, and weight selection, which transfers weights from a teacher model. In this paper, we argue that initialization can introduce an optimization bias that fundamentally shapes training dynamics. We propose conditioned initialization, a principled scheme that initializes attention weights to improve the spectral properties of the attention layer. Theoretically, we show that conditioned initialization can potentially reduce the condition number of the attention Jacobian, leading to more stable optimization. Empirically, it accelerates convergence and improves generalization across diverse applications, highlighting conditioning as a critical yet underexplored area for advancing Transformer performance. Importantly, conditioned initialization is simple to apply and integrates seamlessly into a wide range of Transformer architectures." +4a78f264511a1bfb71e430cbe59e6349206fe6d2,2609.06055,,2026,DriveZero: End-to-End Driving Beyond Human Demonstrations,Hao He,20,,4,"Most end-to-end autonomous-driving systems learn by imitating human driving logs, leaving their learned behavior constrained by the quality and behavioral coverage of the recorded trajectories. This report presents DriveZero, an end-to-end system that learns driving behavior beyond human demonstrations. It decomposes driving into a perception model and an action model, pretrains each in the regime best suited to it, and combines them into one end-to-end planner. The two models call for different learning recipes: perception must understand the world, and benefits from massive and diverse visual data; action must interact with it, and requires closed-loop feedback. On the action side, we introduce DriveRL, a mixed-agent closed-loop reinforcement-learning framework. It converts real driving logs into interactive worlds, where a privileged teacher policy is trained with PPO through closed-loop rollouts. For the perception model, DriveVFM consolidates multiple frozen vision foundation models, including DINOv3, SigLIP2, SAM and Depth Anything V2, into a single backbone from raw images alone, requiring no task-specific annotations. DriveZero then unifies the two: a camera-only planner that distills the frozen DriveRL teacher through its rolled-out trajectories. The goal-conditioned teacher can moreover be queried under augmented driving intents, yielding diverse, goal-consistent supervision that logged data cannot provide. On nuPlan, DriveRL with value-guided test-time action search achieves a mean score of 93.57 across the Val14, Test14-hard, and Test14-random community splits in both non-reactive and reactive modes, exceeding the Log-Replay expert on all three splits. DriveZero achieves state-of-the-art performance on NAVSIMv1, NAVSIMv2 and the closed-loop HUGSIM benchmark without any human trajectory supervision." +d50eac91743c6d3e1a62b87a5de109da60b06e8e,2609.05415,,2026,UniMate: One Unified Model to Animate Diverse Skeletons,Lin-Zhan Mou,7,,2,"Recent advances in automatic rigging now deliver animation-ready 3D assets at scale, yet generating the motion to drive them remains a bottleneck. Existing learned animators are topology-constrained: they rely on category-specific templates or require per-skeleton fine-tuning and reference motions at inference. We present UniMate, a unified foundation model that synthesizes articulated motion for arbitrary skeletons from a rigged 3D asset and a text prompt, with no test-time optimization or per-skeleton retraining. UniMate introduces a topology-aware diffusion transformer, which integrates skeletal topology into attention via three mechanisms: (1) a graph-aware attention bias from pairwise joint relations and geodesic distances; (2) a spectral rotary position embedding generalizing RoPE to arbitrary kinematic trees via the graph Laplacian; and (3) a global topological conditioner attention-pooled from the rest-pose skeleton. We also curate UniML3D, 13,006 motion sequences spanning bipedal, quadrupedal, avian, marine, insectoid, serpentine, and articulated rigid objects with unified canonicalization and text pairing. Trained on this dataset, UniMate outperforms state-of-the-art baselines in quality, generalization, and efficiency, and supports zero-shot cross-topology transfer, in-betweening, expansion, and text-guided editing. Our project page is available at https://linzhanmou.com/unimate/." +04e5ad44902b155634e709a3245a326ba2da878d,2609.05738,,2026,RenderFormer-V2: Neural Rendering with Heterogeneous Scene Primitives,Chong Zeng,5,,1,"We present'RenderFormer-V2', a unified learned transformer-based neural rendering model, complementary to modern physics-based rendering systems, that can handle diverse light-transport effects such as caustics, volumetric scattering, environment lighting, textured and displaced surfaces and out-of-distribution materials without per-scene training or specialized code. RenderFormer-V2 models global light transport as a sequence-to-sequence transformation. Following its predecessor, RenderFormer-V2 also employs a two stage process: a view-independent stage that resolves intra-scene primitive to primitive transport, and a view-dependent stage that transforms the internal neural scene representation into image pixels. Different from RenderFormer, our model employs a novel combined windowed-attention and rendering-informed attention sink in the view-independent stage to improve scalability while maintaining render accuracy. To further improve versatility, RenderFormerV2 supports heterogeneous scene primitives, including environment maps and participating media, and it employs a material encoding independent of the underlying surface reflectance model that encodes material appearance via a novel neural embedding. We demonstrate the versatility of RenderFormer-V2 on a variety of scenes and perform an extensive ablation of the improved attention mechanism." +3e6c99fb6ef4af5c7fbbfc512f17540485940039,2609.03807,,2026,Almost Free State Prediction Separation,John Langford,8,,2,"State--prediction separation (SPS) relieves a language model's hidden state of two competing burdens---summarizing the context and predicting the next token---by splitting the forward pass into a state stream and a prediction stream. The separation works, but it is expensive: the prediction stream is a second pass over the whole backbone, costing $\sim$1.9$\times$ the pretraining FLOPs, and even more in terms of wall-clock time when using a flexible attention mask. This paper makes state--prediction separation almost free. We take the separation to its limit with a free pause token: a prediction stream that writes no keys or values at all and so rides the sequence's existing positions. It improves next-token prediction of a standard Transformer by 2-3 centinats in practice on a 1B parameter model, and because it adds no position it costs nothing at inference---no added context length, no KV cache, no decode steps, and essentially no latency, with the growth in inference flops typically irrelevant as it is not the active bottleneck on throughput. The cost is therefore entirely in training where we use four mechanisms to drive it down: a two-pass split that keeps FlashAttention kernels viable, the $w{=}0$ prediction window, a shared gated FFN that evaluates one FFN per position rather than one per stream, and phasing the separation onto the tail of the run. Together these bring the overhead versus an optimized pretraining pipeline to $1.33\times$ wall-clock while recovering ~94% of the gain compared to SPS, and to as low as $1.09\times$ along a graceful quality/compute tradeoff. Furthermore, the FFN optimization reduces the raw flops required at inference time. The result is an isoflop, isoparameter, and isotoken improvement over standard next token trained transformers." +38624d2075689a3151af1341ad96a6762091ea9b,2609.00746,,2026,Text Capability Loss in Vision-Language Adaptation: An Attention-Sink Diagnosis,Minsik Choi,3,,0,"Fine-tuning a pretrained LLM into a vision-language model (VLM) can erode the backbone's text capability, with the damage concentrated on tasks that require following exact output rules, such as instruction following, chain-of-thought reasoning graded on a strictly parsed final answer, and similar evaluations with strict graders. We trace this gap to attention-sink corruption: VL fine-tuning perturbs the early sink position that anchors a large fraction of attention probability, and how well the base LLM preserves its sink tracks how much of the affected capability survives adaptation. Building on this view, we introduce Sink Strength, a single scalar computed on the base LLM in a few seconds on a single GPU that predicts post-VL degradation without any VL training. It consistently tracks relative degradation across the six VLM-LLM pairs and multiple format-sensitive tasks. Complementing this diagnostic, we find that post-pretraining QK-RMSNorm injection fails to reproduce the protection of native QK-RMSNorm, while several off-the-shelf weight-merging settings fail to recover the lost capability after VL training. These negative results underscore the value of screening backbones with Sink Strength before VL training and narrow the intervention space toward head-selective training-time protection." +b672498982c6c9797c26e27d480b0bff7a744f82,2608.28308,,2026,"Deriving Scaling Laws for OpenEuroLLM Models: Learning Rate, Batch Size and Loss",Niccolò Ajroldi,7,,2,"We study the scaling behavior of learning rate and batch size in pretraining dense large language models on English-prevalent corpora. Beyond scaling jointly optimal learning rates and batch sizes, we investigate their marginal evolution with model capacity and data scale and develop a model that captures these relationships. As we employ a Warmup-Stable-Decay learning rate schedule, we further investigate the gains from learning rate annealing over a broad range of hyperparameters settings, models and data budgets, and whether the optimal learning rate and batch size transfer between the stable and decay phases. Finally, we characterize the dependence of loss on model capacity and dataset size, evaluating recently proposed scaling forms that explicitly model their interaction. We find these approaches particularly effective at capturing both undertraining and overtraining regimes across our experiments. This study establishes a first baseline and scaling procedure for the development of future OpenEuroLLM models. We open-source the complete collection of pretraining runs used in this study." +032abd5102a3cdea5898579dc8c6363de85ee032,2608.26973,,2026,Squeezing More from Limited Data with Recursive Transformers,Serdar Gülbahar,3,,0,"Pre-training under limited data requires a different view of scaling than web-scale language modeling. With a fixed data budget but relatively abundant compute, increasing parameter count helps only up to an optimal scale; beyond that point, models overfit and generalization worsens. We study this behavior across 10M-100M word pre-training budgets, two corpora, and multiple downstream evaluations, and find that optimal size depends strongly on both the data budget and the downstream target. We argue that standard Transformers scale down poorly to this setting, because embeddings consume a large fraction of the parameter budget and per-token computation is tied to representational capacity. To address this coupling, we study recursive Transformers, reusing a shared block across depth to scale compute, together with factorized embeddings to reduce vocabulary-map parameters. We train three recursive models and find that they outperform standard Transformers at 10M and 100M words, while remaining competitive with BabyLM Challenge 2025 winners." +e568f4a2aea628ba01ea12726a2f1465f28555e6,2608.24814,,2026,Effective Learning Rate Governs Loss Dynamics in Language Model Pretraining,Zi-Han Liu,7,,5,"We uncover ELR collapse in language model pretraining: learning rate (LR) and parameter norm govern loss dynamics primarily through their ratio, the effective learning rate (ELR). When ELR is matched across runs, their loss trajectories collapse throughout training despite substantially different LRs and parameter norms. Across optimizers, architectures, datasets, and model scales, mean collapse errors are typically a few x 10^-3, below the seed-to-seed variation measured in a representative configuration. Systematic ablations identify normalization design and the timescale of LR-norm variation as key determinants of collapse precision. Controlled interventions further show that weight decay and Hyperball shape loss dynamics primarily through the ELR schedules they induce. Replacing LR with ELR enables a fitted functional scaling law (FSL) to transfer across norm-control methods. The resulting ELR-based FSL also explains delayed acceleration, a recurring effect of norm control. Together, these results establish ELR as a common coordinate linking LR scheduling, norm control, and loss dynamics." +6be4eeea4a7860b32ea4ba3ea631f0c332b89987,2608.21174,,2026,From Attention Masks to Inert Zero-Vector Tokens: OAttention and O-Closure for Token Dynamics,Heyang Gong,1,,0,"Attention masks are relation-level controls: they specify which query--source pairs may interact. They do not provide a representation-carried token state that is non-participating at the attention boundary. We assign each token hidden carrier \(h_i\) an active-presence coefficient \(p_i=\lVert h_i\rVert^2/(\tau+\lVert h_i\rVert^2)\). The same coefficient has two roles: it gates information emitted by token \(i\), and it determines the mass with which token \(i\) enters computations shared with other tokens. OAttention is the support-coupled attention realization of this rule. It gates the receiver output by \(p_i\) and weights source \(j\) by \(p_j\) in both the attention numerator and partition, while retaining the standard score, visibility relation, exponential competition, and value aggregation. This makes the zero-vector token a zero element and yields exact null-receiver, null-source insertion, self-attention insertion, and empty-support properties. The same token-level presence gives local O-components (OFFN, ONorm, and OInject), presence-weighted OStandardize, the O-Closure law \(M(H\oplus0)=M(H)\oplus0\), and an OTransformer by residual and compositional closure. The canonical operator is checked by contract tests and a GPU evaluation. In a zero-fine-tuning retrofit of a cloned pretrained TabPFN v3 regressor, calibrated hidden-carrier OAttention and Full-O variants change mean RMSE by $+0.088\%$ and $+0.177\%$, respectively, over 18 matched dataset--seed cases. A two-block ablation shows that OAttention alone does not preserve a NULL state through ordinary host components, whereas the OTransformer path does. These are scoped tests of exactness, active-path compatibility, and compositional necessity; they do not establish universal no-loss, arbitrary-host closure, learned attraction to the origin, or a general semantics for missing values." +bcf914375c30492a0e14e8e289eb1d9794e887bd,2608.19863,,2026,Listening Forward: Next Patch Embedding Prediction Enables Scalable Audio Learners,Umberto Cappellazzo,4,,0,"Self-supervised learning (SSL) has driven substantial progress in audio representation learning, though existing methods have increasingly relied on elaborate pre-training recipes to reach competitive performance. A markedly different pre-training philosophy underpins the most influential progress in language modeling and, more recently, in visual representation learning: rather than train encoders as static feature extractors, models are trained to predict the next element, a discrete token or a continuous embedding, from the preceding context. Autoregressive prediction thereby provides a unified pre-training interface that transfers across modalities, compelling the model to learn the underlying data distribution. We ask whether such a simple causal paradigm can yield strong audio learners, given that audio's temporal structure makes autoregressive prediction of patch embeddings a natural fit. We introduce NAPE (Next-Audio-Patch-Embedding prediction), a self-supervised framework in which a causal Transformer predicts each next patch embedding of a log-mel spectrogram from the previous ones, using causal masking and stop-gradient as its sole training signal. The design is intentionally minimalist, avoiding reconstruction decoders, acoustic tokenizers, student-teacher setups, and auxiliary regularization losses. Across six audio and speech benchmarks, NAPE achieves state-of-the-art fine-tuning performance on several tasks, scales consistently across encoder sizes, and yields strong linear-probing results. NAPE also produces structured attention patterns without explicit supervision." +d6caee81c9e72b71a18529aa54409f2ca8661f6e,,10.1117/12.3101171,2026,Performance of attention-based deep-learning models in Shack-Hartmann wavefront sensor measurements,Matias Marambio-Jimenez,7,Astronomical Telescopes + Instrumentation,0,"A systematic performance evaluation of attention-based deep-learning models for wavefront reconstruction in Shack-Hartmann wavefront sensor (SHWFS) systems is presented. While deep learning has been extensively applied to pyramid and focal-plane wavefront sensing, its application to standard SHWFS configurations remains relatively underexplored. A simulation dataset consisting of 3.5 million open-loop uncorrelated turbulent phase and image pairs modeled after the Subaru Telescope pupil was generated. The dataset spans four pupil-sampling cases (8×8, 16×16, 20×20, and 32×32 subapertures) under realistic conditions. A custom Vision Transformer (ViT) which incorporates features such as rotary positional embeddings, register tokens, and a cross-attention decoder, is benchmarked against a fully convolutional U-Net and a classical Maximum A Posteriori (MAP) linear reconstructor. The results show that the ViT architecture yields the lowest overall residual phase variance across all tested subaperture configurations, demonstrating a robust capability to model global spatial correlations and spot-to-spot cross-coupling across the pupil. While the U-Net performs competitively by capturing localized features, it exhibits ∼ 1.5 times more residual variance per mode than the ViT, which is relatively significant in high-order modes. This study represents a key step toward implementing attention-based models for high-order wavefront sensing and telemetry-assisted data reduction in next-generation instrument suites like ULTIMATE Subaru." +928b977bfc1df405f7ad90f0ceeedb492ecbae8c,2608.17286,,2026,Abra: Scaling Diffusion Image Training,Kyle R. Chickering,8,,3,"Compute-optimal scaling laws guide the training of frontier language models yet remain largely unexplored for visual generation. We present a systematic scaling law study for text-to-image diffusion models using Abra, a controlled family of flow-matching transformers trained across three orders of magnitude worth of compute ($10^{19}$ to $10^{22}$ FLOPs), reaching significantly larger compute budgets than previous works. We demonstrate that diffusion models scale just as predictably as language models but require far more data to train optimally: compute optimality occurs at approximately $200$ image tokens per parameter, ten times the Chinchilla compute-optimal prescription for LLMs. We show that unlike language models, diffusion models are robust to overtraining and that practitioners should err on the side of more data rather than a larger model. Finally, we show that this predictability extends beyond training loss to generative quality metrics, optimal CFG settings, representation quality, and even the shape of the training curves, which collapse onto a universal form." +902b5343fce9bd5216254ac1d10d365316080c81,2608.11519,,2026,FLARE++: Low-rank attention with dynamic attention routing,Vedant Puri,3,,0,"Full self-attention is a strong token mixer for PDE surrogates on irregular domains, but its quadratic cost limits its use on large problems. Latent-space attention methods such as PerceiverIO, Transolver, and FLARE (Fast Low-rank Attention Routing Engine) avoid that cost by routing attention among $N$ tokens through $M\ll N$ learned latents. They compress by dot-product matching of the input tokens against $M$ learned query tokens or projection weights: once trained, the same learned templates serve every input. We remove this restriction with FLARE++, a low-rank attention architecture with input-conditioned routing queries. FLARE++ uses FLARE's own encoder to map the $N$ input tokens to $M$ query tokens, which correct the learned queries. The adapted queries then determine how that same input is compressed and redistributed. This preserves FLARE's explicit low-rank factorization and linear $\mathcal O(NM)$ complexity, and expresses the complete routing operation with standard scaled dot-product attention (SDPA) calls alone. We also provide a multi-GPU context-parallel implementation that shards input tokens across devices without ever gathering the full token sequence on one of them. FLARE++ reduces FLARE's error by $25\%$ on average across five standard PDE benchmarks, achieving the lowest errors among the efficient models compared. The gains persist on industrial-scale DrivAerML aerodynamics and on Long Range Arena, where average accuracy rises by $3.5\%$ over fixed-query FLARE. Code is available at https://github.com/vpuri3/FLARE.py." +57c4a60449eabd5b7ff41cc328171121b9728486,2608.11015,,2026,Sona Technical Report,Sona Team Alexandr Udeneev,34,,0,"We introduce Sona, a single-model generative recommender for Yandex Music. In an online A/B test, Sona replaced the entire production cascade, comprising more than 15 candidate generators followed by pre-ranking and ranking models that consume hundreds of features, including signals from large transformer models such as Argus and target-attention scorers, while significantly improving key engagement metrics. The architecture of Sona unifies candidate generation and ranking around a shared user representation. Its encoder transforms the user's chronological sequence of logged engagement events into hidden states consumed by both the autoregressive decoder and the Ranking Module. The next-token-prediction and distillation objectives jointly update the encoder, coupling generation and ranking through the same user state. Neither Sona nor its Teacher Ranker uses hand-engineered features; both operate on logged event fields and learned item representations. In the final Sona configuration, the larger teacher supplies ranking targets during training but is absent from serving, leaving the encoder, decoder, and Ranking Module as a single deployed model. We evaluate Sona in an online A/B experiment using live traffic from My Vibe on smart speakers, one of Yandex Music's largest recommendation surfaces. Relative to the production control, Sona produced statistically significant uplifts of 4.53% in Active Users, the primary metric, 6.30% in Total Listening Time, and 11.42% in Likes. These effects were incremental to improvements retained from preceding deployments. The Active Users uplift was 2.35 times the increment previously delivered by Argus, the strongest model deployed on this surface before Sona. These results show that a single jointly trained model can replace a mature multi-stage recommendation cascade while improving recommendation quality on live traffic." +f0cbd4e7bb6d0b7d83fadecd10b8508cf19ef21f,2608.03158,,2026,Surface Keypoint Representation for Multi-Object and Articulated Human-Object Interaction Generation,Xiaogang Peng,7,,2,"Daily activities require humans to coordinate whole-body motion with the motion of surrounding objects. Despite recent progress in human-object interaction (HOI) generation, most existing methods assume interactions with a single rigid object and do not extend well to scenarios involving a variable number of objects or articulated objects with diverse joint mechanisms. We propose surface keypoint trajectories as an object motion representation: for each rigid component, whether a standalone object or one part of an articulated assembly, we track a small set of non-collinear surface points over time. This representation handles multi-object coordination and diverse articulation mechanisms directly from point dynamics without requiring explicit joint-type specification. To model when and where each body region contacts each object, we introduce a spatio-temporal contact distance field that extends distance-based contact modeling to whole-body, multi-object, and articulated settings. We factorize HOI generation into three stages: generating object motions from text or waypoints, predicting the contact distance field, and synthesizing whole-body motion with contact-guided optimization. Experiments on ParaHome, HIMO, ARCTIC, and OMOMO demonstrate better or comparable performance to existing methods across single-object, multi-object, and articulated interaction settings." +ed30f8b49273ff553353a7f43a067ce19beff13b,2608.03624,,2026,LoopMTP: A looped transformer guided by latent multi-token prediction,Behzad Shomali,5,,3,"Looped transformers have emerged as a parameter-efficient alternative to scaling depth for strong reasoning. By reusing one stack of layers across $T$ iterations, they attain the effective depth and reasoning capabilities of larger models at a fixed parameter count. Yet existing approaches suffer from latent overthinking and undifferentiated computation, largely because intermediate representations receive no guidance across loops. Multi-token prediction (MTP) supplies exactly the dense, forward-looking supervision the loop is missing. We propose \textsc{LoopMTP}, which links the two through a structural correspondence in latent space: a model that loops $T$ times can anticipate $T$ future tokens. \textsc{LoopMTP} realizes this by softly aligning the hidden state of loop $t$ with the embedding of the token $t$ steps ahead, while a lightweight gate preserves useful information across iterations. \textsc{LoopMTP} improves average accuracy by up to 8.1\% (relative) over the non-looped baseline, with training remaining stable for up to 15 loops." +1dcfd96d0e2ce30134899bd78053ea6741394e7c,2608.01920,,2026,P-MUSE: Prompt-MIDI-Optional Model for Unified Instrumental Music Synthesis and Editing,Chong Jing,6,,2,"MIDI-to-Music system renders the melody and rhythm of a target MIDI sequence into musical segment while cloning instrument timbre from a prompt recording. Existing systems typically adopt one of two distinct paradigms: conditional generation with prompt audio alone, which remains applicable when aligned prompt MIDI is unavailable, and In-Context Learning with paired prompt audio and MIDI, which exploits cross-modal alignment for stronger control on MIDI following and timbre similarity. We introduce P-MUSE, an instrumental MIDI-to-Music framework that unifies both paradigms via a multi-stage Curriculum-Learning supporting prompt-MIDI-optional inputs. P-MUSE further unifies music generation and local editing through a shared fill-in-the-middle formulation. Grounded in theoretical analysis and empirical study, we propose a phase-aware classifier-free guidance scheduling principle for Transcription-to-Audio systems, alongside a Tail-Drop strategy. Finally, to advance research in this field, we establish the first comprehensive benchmark, covering various prompt modes, generation/editing tasks, and four representative instruments: piano, guitar, bass, and drums. Demos are available at https://p-muse.github.io/." +409892b100dfd82428951144aba6e5cbfb3a5361,2608.02602,,2026,AURORA-LM: Autoencoding Unified Representation for Continuous-Latent Diffusion Language Modeling,Jia-Jun Liang,15,,8,"Language remains an outlier in generative modeling: while images, video, and audio are increasingly modeled in continuous latent spaces, text generation still relies predominantly on discrete tokens. Existing continuous language models either inherit embedding spaces not designed for joint generation and decoding, or compress autoencoded latents to ease diffusion, sacrificing token-level fidelity. Instead of simplifying the representation to suit the generative model, we preserve a high-capacity, decodable text latent and design the diffusion model to learn its distribution directly. We introduce AURORA-LM, a continuous-latent diffusion language model that separates the construction of a decodable text representation from the modeling of its distribution. A Query-based Encoder-Decoder organizes text into a high-capacity, prefix-aligned latent sequence, and a Block-causal Diffusion Transformer learns its distribution through flow matching, generating blocks left to right while denoising positions within each block in parallel. Because such a latent is harder for diffusion to model, AURORA-LM restricts only the noisy-input pathway while retaining the full clean-latent prediction target, accommodating full-width latents without reducing decoder-facing capacity. We further calibrate the noise-level distribution to the latent width, and introduce self-trajectory consistency to bridge independently sampled training noise and iterative denoising at inference. AURORA-LM achieves the strongest performance among evaluated continuous and diffusion-based language models on OpenWebText free generation and XSum summarization. Scaling to 1B parameters with about 1500 EFLOPs of total compute yields further gains, surpassing a larger publicly released latent-diffusion language model under a matched evaluation protocol. All experiments are conducted on Ascend NPUs." +9653176d4dafd012f0b04775889b80cc7b8c6b91,2608.02091,,2026,"One QK Channel, Many Sources: Tracing Low-Precision Attention Collapse",Shu-Xiao Xie,6,,0,"A bfloat16 transformer can train normally, then collapse abruptly. Prior work links collapse to structured attention errors and shows QK normalization disrupts their compounding. Distinct low-precision errors trigger the same collapse, leaving unclear whether each needs a fix at its source or one shared route can be blocked instead. We isolated the fault behind a reproduced GPT-2-class collapse to the streaming-softmax accumulator, where an fp32 streaming core repairs it, and turned it into an assay for moving a controlled error across sources. Using it, we found that errors placed outside attention still drove the same QK spectral runaway, and that correcting only QK kept training stable while the fault stayed active. This is a source-channel dissociation: fault source is not failure channel. It held across tested architectures and scales, and reproduced on a second GPU architecture. As a causal probe, projecting each update off the current QK weights'leading three singular directions held the query projection's largest singular value to 11.1, whereas removing equal energy elsewhere left it at 237: the QK channel causally drives the early runaway. What lets the injected error in is temporal sign-coherence, its per-head sign persisting across steps, not aggregate deviation; once inside, the runaway shows as attention-logit saturation. QK-Guard, a dormant controller, tests this by switching on parameter-free QK normalization at the first monitored threshold crossing. On the runs designated for this test, the QK-local action prevented the failure of each matched or same-configuration unguarded run; on plain GPT-2, all 12 final train and validation losses were within 0.03 nat of same-configuration always-on QK-norm, and both methods ran 60k steps without collapse. Intervention at the QK locus therefore suffices in place of a fix at each source." +b086e0018a7af6940362c0f5b5b81c4174044f48,2608.02365,,2026,Faster-WAM: Do World Action Models Need Deep Action Modules?,Liheng Ma,11,,15,"World Action Models (WAMs) build on pretrained video models, whose representations are grounded in physical dynamics and provide a natural basis for action prediction. Despite this natural foundation, many WAMs still rely on deep, parameter-heavy action-prediction modules that incur high inference latency and may overfit to limited robot demonstrations, restricting their real-world applicability. In this paper, we advocate a world-model-centric principle that concentrates capacity and computation in the video world model, while a lightweight action expert translates the backbone's representations into executable robot actions. We realize this principle through three key choices: Dock of Transformers (DoT) with Lite KV-Fusion to give the shallow, lightweight action expert access to representations from all video layers; world-model-only conditioning of the action expert; and retracted 1D-RoPE for positional alignment between video keys and action queries. We test this principle using Faster-WAM, a world-model-centric WAM with only a single-layer action expert. Despite this restriction on action-specific computation, Faster-WAM achieves competitive control performance on LIBERO and RoboTwin~2.0 without additional embodied pretraining. It provides approximately $3.7\times$ and $1.3\times$ inference speedups over Fast-WAM and $\pi_{0.5}$, respectively. Consistent with its world-model-centric design, Faster-WAM demonstrates stronger generalizability under distribution shifts: the same LIBERO-trained policy achieves $78.3\%$ success on LIBERO-Plus, exceeding Fast-WAM and LingBot-VA by $26.8$ and $8.8$ percentage points, respectively. Finally, real-robot experiments demonstrate success rates comparable to Fast-WAM, with substantially lower inference latency and shorter task-completion times." +8af2d906621c35c4dfdaef1a41339eb0a342dc9a,2608.01284,,2026,Training nGPT,I. Loshchilov,2,,0,"The normalized Transformer (nGPT) realizes hyperspherical representation learning by constraining model parameter vectors and activation vectors to the unit hypersphere. In this paper, we describe a practical training recipe for nGPT and evaluate it on modern hybrid Mamba-2--Transformer Mixture-of-Experts (MoE) models. The recipe introduces Logit Gradient Preconditioning, Logarithmic Learning Rate Decay, GatedAdamW, angular update control, and optional exploration mechanisms. Compared with an unnormalized model of the same hybrid MoE architecture trained with AdamW, the 30B-total-parameter nGPT model reaches the same validation loss using approximately half as many training tokens. The recipe scales across the models considered, which contain up to 30B total parameters." +1b6f1a575a82efa1236e17d17f0d1453c3e919c5,2607.27308,10.48550/arXiv.2607.27308,2026,ZUNA1.1: A more flexible EEG foundation model for Denoising and Super-resolution,Christopher Warner,4,arXiv.org,0,"We introduce ZUNA1.1, a 380M-parameter diffusion autoencoder for flexible EEG signal reconstruction. ZUNA1.1 is capable of reconstructing variable length sequences of up to 30s, with an arbitrary number of EEG channels at arbitrary scalp locations, and can reconstruct arbitrary temporal intervals within channels in addition to reconstructing entire channels. We demonstrate that ZUNA1.1 performs at least on par with our earlier ZUNA1 model, while being far more flexible and capable of handling a wide range of reconstruction tasks. ZUNA1.1 continues to substantially outperform standard EEG denoising and reconstruction methods such as spherical spline interpolation, which is ubiquitously deployed in the MNE package. The ZUNA1.1 model is released open source under the permissive Apache 2.0 license." +5fc18c66965fc9b4ccf3ef17793d2e8de219acdc,2607.25357,10.48550/arXiv.2607.25357,2026,Raven: High-Recall Sequence Modeling with Sparse Memory Routing,Arshia Afzal,5,arXiv.org,10,"Long-context recall in linear-time sequence models highlights a tradeoff in how they write to memory. State-based linear models, such as state-space models (SSMs) and linear Transformers, write densely, updating the entire state for each newly arrived token, which leads to interference and makes specific past tokens hard to recover. Sliding-window attention (SWA) exhibits the opposite behavior: it writes sparsely by storing explicit token representations, but only within a fixed window, so recall drops once the relevant token is evicted. Interpolating between these models, we introduce Raven, a linear-time sequence model that maintains a fixed set of memory slots and, at each step, decays and updates only a selected subset via learned, input-dependent routing. This lets Raven mitigate SWA's position-based overwriting and hard eviction while reducing interference from dense state updates in SSMs, thereby preserving long-range content much more effectively. Across recall-intensive benchmarks, Raven is competitive with or outperforms prior linear-time baselines, achieving strong long-context recall where both SWA and SSMs sharply degrade. It remains effective when extrapolating to context lengths as large as 16x its training length, with similar gains in hybrid architectures." +ad25b88d2eb9427de8b760f688ab73db2ac208f8,2607.18363,10.48550/arXiv.2607.18363,2026,A Controlled Study of Attention-Only Transformers,Henry Ndubuaku,8,arXiv.org,1,"Feed-forward networks hold two thirds of a transformer's non-embedding parameters, yet the architecture has not received a necessity test that controls parameters, compute, and depth at once. We pretrain attention-only decoder transformers (Simple Attention Networks, SANs) against standard transformers matched separately for parameter count, training FLOPs, and depth (2 to 48 layers), for up to 105B tokens at 6M to 87M parameters. Deleting feed-forward layers in place is costly: the standard transformer leads by 0.47 nats at matched depth and 0.26 nats at matched FLOPs. Reallocating the freed budget into attention depth closes the gap: at matched parameters the difference is 0.006 nats (0.27 percent of loss), reproducible to one part in ten thousand across seed pairs, shrinking across 5B, 30B, and 105B budgets, and holding near 0.02 nats across a 29x size range. Three measurements localize the remaining gap to parametric recall: attention-only models are better on context-grounded answers and worse where knowledge must come from weights. Weight spectra show why: routing matrices (Q/K) crystallize early, content matrices accumulate rank slowly, and removing feed-forward layers relocates this accumulation to the attention output projection. QK-normalization, not feed-forward layers or residual gating, keeps 48-layer attention-only stacks trainable. The deficit concentrates on low-context query prediction and localizes there entirely by the largest budget. A pre-registered test confirms the account: it predicts a 0.02 to 0.05 nat gap on knowledge-dense web text; a matched pair trained on fineweb-edu measures 0.040. Within the tested regime, attention does the rest." +3d4598c71a7d5c07aa6cd29d70e765ea9f524241,2607.13022,10.48550/arXiv.2607.13022,2026,A Shortcut to Statistically Steady-State Turbulence with Flow Matching,Gianluca Galletti,8,arXiv.org,0,"Many nonlinear physical systems exhibit an initial transient phase in which perturbations grow before nonlinear interactions lead to a statistically steady state. While this saturated regime is of primary interest, direct numerical simulations must resolve the full transient dynamics before reaching it, incurring significant computational cost. In Computational Fluid Dynamics, reduced-order approaches such as Large Eddy Simulation mitigate computational cost by modeling small-scale dynamics, enabling tractable approximations of turbulent flows. In contrast, for systems such as gyrokinetics, comparably effective closures for the full dynamics are not generally available, and high-fidelity simulations remain necessary. Existing surrogate modeling approaches for these systems are autoregressive, hence they suffer from accumulating error. We instead propose to bypass explicit time evolution by directly modeling the distribution of saturated states under an ergodicity assumption, stating that ensemble averages over samples are equivalent to time averages of a single long simulation. We introduce GyroFlow, a latent generative model that directly estimates steady-state statistics of gyrokinetic turbulence in 5D phase space, without resolving the transient phase. GyroFlow generates saturated snapshots from noise, conditioned on dimensionless operating parameters and outperforms autoregressive, reduced-order, and other generative approaches, while providing substantial speedup. To evaluate generation quality we propose FGyD, a distributional metric computed in the latent space of a pretrained gyrokinetic model, and show that it correlates with downstream flux accuracy and solver convergence. Finally, GyroFlow can be used to warm-start the numerical code used to produce the data." +40b73dba3f28ae798e018641c687b98f242d0e7f,2607.13193,,2026,Generating synthetic evolution of turbulent flames with an experimental data-based spatiotemporal diffusion model,Amrit Tarur,2,,0,"In this study, a conditional diffusion model -- a class of generative machine learning models -- is developed to generate synthetic, experimental data-based trajectories of turbulent flames. Generated experimental data corresponds to simultaneous field measurements, namely OH planar laser-induced fluorescence (OH-PLIF) fields and multi-component particle image velocimetry (PIV) fields, for attached and detached flame states in a swirl combustor configuration. This is done using an x-prediction flow matching framework combined with a pixel-based spatiotemporal transformer, which is capable of generating entire spatiotemporal slabs containing synthetic flame evolution at inference time, conditioned on the flame regime. Using this framework, synthetic flames were found to preserve key flame features and statistical consistency across space and time, particularly at the large scales -- deviations at high temporal frequencies and small spatial length scales were found to depend on the time-span of the generated space-time slabs. An extrapolation task of transition synthesis is also conducted, in which the conditional diffusion model is used to synthesize spatiotemporally coherent flame transitions (flame liftoff and reattachment) unseen by the model during training. This was accomplished using a model for the denoising transition velocity that relies on time-varying linear combinations of attached and detached denoising velocities, leading to an approach that (a) allows for control of the generated transition directions and timescales, and (b) retains sample-to-sample variability in the generated transitions in the process. Overall, this study provides a promising pathway for the utilization of experimental data-based generative models as a new means of data exploration in data-sparse environments, complementing both experiments and computational fluid dynamics-based approaches." +8b7387f357e6436c7c492206bbf24466f7d795bf,2607.04068,10.48550/arXiv.2607.04068,2026,UniSGR: Unified Framework for Semantic ID Generation and Ranking,Jia-Wei Sun,7,arXiv.org,3,"Recommendation systems play a pivotal role in modern e-commerce platforms. While generative retrieval has emerged as a promising paradigm for alleviating the limitations of multi-stage cascade architectures, existing methods still struggle with fine-grained multi-objective ranking. To bridge this gap, we propose UniSGR, a Unified framework for Semantic ID Generation and Ranking. UniSGR adopts a two-stage training paradigm: a multi-scenario pre-training stage that learns from mixed business-scenario data, followed by a scenario-specific alignment stage that jointly optimizes Value-Aware Parallel Multi-Token Prediction (VA-PMTP) and a unified multi-objective ranking module. To better align generation with downstream ranking, we introduce Task-Aware Tokens (TAT) guided by Funnel-Aware Contrastive Learning. Furthermore, we propose Semantic Tree Attention with Reorganized KV cache (STARK), an inference strategy that removes key efficiency bottlenecks in conventional beam search. Extensive offline experiments on a large-scale e-commerce platform demonstrate the effectiveness and scalability of UniSGR." +3718f64abd0c2c0241cdd2ab3f10f2b382222aa4,2607.04064,10.1109/OJSP.2026.3711530,2026,Speaker-Disentangled Chunk-Wise Regression for Syllabic Tokenization,Ryota Komatsu,4,IEEE Open Journal of Signal Processing,0,"Unsupervised syllabic tokenization aims to learn discrete syllabic tokens that capture latent linguistic content-related structure from raw speech. Recent syllabic tokenization methods employ teacher-student distillation of the pretrained HuBERT to organize latent speech frame representations into syllabic segments. However, when trained with an utterance-level cross-entropy objective, the model predicts speaker identity rather than linguistic content, thereby compromising the purity of syllabic tokens. To address this problem, we propose a speaker-disentangled syllabic tokenizer that regresses speaker-perturbed student representations toward clean teacher targets within fixed-length chunks. Experimental results demonstrate that our proposed method achieves state-of-the-art performance in syllable boundary detection and syllabic segment clustering. Moreover, a speech language model trained on our syllabic tokens achieves a 7% relative improvement in syntactic and semantic understanding over the phone-level SpiRit-LM." +bae8d2f66cd289f690db7d3f89e2951e34229d8e,2607.02770,10.48550/arXiv.2607.02770,2026,Gemma 4 Technical Report,Gemma Team Sherif El Abd,300,arXiv.org,311,"We introduce Gemma 4, a new generation of open-weight, natively multimodal language models in the Gemma model family. Designed to advance compute efficiency and reasoning, the Gemma 4 model suite features dense and Mixture-of-Experts architectures, ranging from 2.3B to 31B parameters. Alongside improved vision and audio encoders for all model sizes, we propose a unified, encoder-free architecture for our 12B model, which ingests raw audio and image patches. Furthermore, we integrate a thinking mode, enabling Gemma models to generate reasoning traces prior to responding. We improve inference speed, memory, and compute efficiency, as well as long-context abilities through critical design choices. Gemma 4 establishes a leap in performance across STEM, multimodal, and long-context benchmarks, and rivals larger, frontier open models in human-rated tasks." +6caaf08a47491189c04c8f53fe0d234a98c720ac,2606.32036,10.48550/arXiv.2606.32036,2026,PointSplat: Compact Gaussian Splatting via Human-Centric Prediction,Yujie Guo,10,arXiv.org,0,"Producing 3D human representations from input views on the fly is essential for immersive live streaming systems, where representation compactness is as critical as high fidelity given limited computational power and transmission bandwidth. Although recent feed-forward reconstruction methods achieve impressive quality through the view-centric prediction of 3D representations, they repeatedly encode the same subject content across multiple views, leading to significant inter-view redundancy. Our key insight is to perform predictions directly in 3D space, enabling the network to learn and produce a highly compact representation. To this end, we propose PointSplat, a novel human-centric approach that directly infers Gaussian primitives from an input point set. The proposed method first estimates a coarse geometric proxy and performs ray casting to prune redundant points and establish explicit 2D--3D correspondences. Subsequently, it employs a Point-Image Transformer to fuse appearance and geometry features, predicting Gaussian attributes in a single forward pass. This design restricts predictions to foreground regions of interest, substantially reducing the total number of Gaussians while improving novel-view rendering quality. Extensive experiments demonstrate that PointSplat achieves higher efficiency and quality while exhibiting strong robustness to variations in view count and image resolution across multiple datasets." +7a2cacbd05be8ef803bc233762e5c458ee7f4a2e,2606.31248,10.48550/arXiv.2606.31248,2026,Scaling Storm-Resolving Atmospheric AI Simulation to the Entire Planet,Ze-Yuan Hu,12,arXiv.org,2,"Kilometer-scale convection shapes precipitation extremes, tropical organization, and cloud feedbacks, but most global atmospheric models approximate these processes at 25-100 km resolution. Global storm-resolving physics models resolve convective systems explicitly, but at a cost -- roughly one MWh per simulated day on exascale supercomputers -- that limits long-duration simulation. We introduce STRATA (Storm-resolving Tile-based autoRegressive Atmosphere Transformer Architecture), the first autoregressive AI emulator for global storm-resolving atmospheric dynamics. STRATA is trained on the highest-resolution atmospheric dataset yet used for global AI emulation: 17 days of SCREAM physics-model output at 4.9-km resolution (~25 million grid cells) sampled every 10 minutes. Our central premise is that on 10-minute timescales atmospheric dynamics are predominantly local, so training on small spatial tiles trades scarce global temporal samples for abundant local spatial samples and enables global rollout via overlapping-tile blending. STRATA combines 3D patch embedding and local 3D neighborhood attention, a novel Stereographic Rotary Position Embedding (StereoRoPE) for grid-invariant encoding, and a pixel-space de-aliasing decoder that suppresses patch-scale rollout artifacts. An iso-FLOP scaling study reveals that km-scale emulation requires ~10x more FLOPs per grid point than coarse-resolution AI weather models, consistent with the higher information density of convective-scale dynamics. Trained on only 17 days of data, STRATA produces stable 24-hour global rollouts with realistic km-scale dynamics across diverse regimes, though large-scale biases develop with lead time. It achieves 48 simulation days per megawatt-hour -- about 50 times better energy efficiency than the SCREAM physics model -- and 741 simulated days per wall-clock day at 512 H100 GPUs. Code and dataset are publicly available." +0c016749121e0bab374152c88a2db2ab9ea25304,2606.29059,10.48550/arXiv.2606.29059,2026,Flow Matching in Feature Space for Stochastic World Modeling,François Porcher,4,arXiv.org,6,"World modeling requires forecasting uncertain futures while preserving information useful for downstream perception. Existing visual world models often struggle to satisfy both goals: VAE-based stochastic models operate in low-dimensional reconstruction latents, which can limit perception performance, while deterministic predictors using strong pretrained features collapse multimodal futures into a single blurry mean. In this work, we propose FlowWM, a stochastic world model that performs flow matching directly within pretrained feature space (e.g., DINOv3). This is challenging because pretrained features are substantially high-dimensional, making standard diffusion recipes suboptimal. To address this, we investigate the design choices needed for feature-space flow matching and introduce a differentiable one-step projection mechanism that enables efficient training with temporal consistency and task-driven objectives. We evaluate FlowWM on two benchmarks: a synthetic benchmark for systematic evaluation of accuracy and diversity, and a real-world benchmark FuturePerception. FlowWM improves perception performance, mode coverage, and horizon robustness, validating our proposed design for stochastic world modeling in high-dimensional feature spaces." +41c5d141b6eb32aa30528590dc583f620d2b97d6,2606.27364,10.48550/arXiv.2606.27364,2026,PhysiFormer: Learning to Simulate Mechanics in World Space,Yi-Ming Chen,3,arXiv.org,2,"We present PhysiFormer, a diffusion transformer for physically-plausible 3D object motion. Unlike video world models that operate in view-dependent pixel space, PhysiFormer represents objects as 3D meshes expressed in world coordinates. Given the initial vertex positions and velocities, as well as object material type, rigid or elastic, the model samples future vertex trajectories. While related neural physics approaches build on ad-hoc latent spaces or explicitly enforce rigidity and causality, PhysiFormer shows that excellent results can be obtained without any such inductive biases, by casting vertex trajectory prediction as a single denoising diffusion process directly in world coordinates. The probabilistic formulation captures uncertainty in the learned dynamics, enabling diverse plausible futures from initial conditions, making this framework potentially useful for applications with unobserved uncertainty. The model features attention factorised over time, space, and objects for efficiency, enabling permutation-invariant multi-object reasoning without needing explicit object encoding. Trained on over 100k simulated trajectories, PhysiFormer generates rigid and elastic mechanics, and generalises to mixed-material settings, unseen real-world geometries, and larger object counts. It substantially outperforms autoregressive baselines in trajectory accuracy, rigidity preservation, and momentum-based physical consistency. Our results position coordinate-space diffusion as a promising step toward view-invariant, geometry-aware world modelling for robotics, graphics, and physical design. Visualisations, code, and models are available at https://yimingc9.github.io/physiformer." +f063ad0ced3448b9c78b906a0c3384be57f58ef8,2606.27375,10.48550/arXiv.2606.27375,2026,"Scalable Behavior Cloning with Open Data, Training, and Evaluation",Arthur Allshire,18,arXiv.org,16,"We introduce ABC, a fully open-source stack for manipulation with behavior cloning. At its core is ABC-130K: the largest open-source teleoperation dataset to date, featuring 3,500 hours of data spanning over 130K episodes across 195 diverse tasks. Furthermore, we open-source our accessible hardware setup, training infrastructure, and simulation pipeline. We also release 400 hours of sim-teleop data and provide a co-training recipe that produces correlated simulation and real-world evaluation, offering a reliable proxy for ablating model-design and training decisions before costly real-world evaluation. We explore various training recipes and compare common architectural choices for Diffusion Transformers (DiT) and Vision-Language-Action (VLA) models, grounding our findings in real-world evaluations. The resulting policies successfully execute dexterous tasks such as box folding and extracting credit cards from wallets. By providing a reproducible toolkit, we aim to place researchers on an equal footing, establishing the necessary foundation to learn the ABCs of Behavior Cloning together as a community." +7dd4d28614cb2d482e0463267692877c958f8e6a,2606.27326,10.48550/arXiv.2606.27326,2026,Hallucination in World Models is Predictable and Preventable,Nicklas Hansen,2,arXiv.org,10,"Modern generative world models render increasingly realistic action-controllable futures, yet they frequently hallucinate: rollouts remain visually fluent while drifting from the ground-truth dynamics. We hypothesize that hallucination concentrates in low-coverage regions of the state-action space, where lightweight data-centric signals can both detect it and guide mitigation. To test this, we introduce MMBench2, a 427-hour, 210-task dataset for visual world modeling with ground-truth actions, rewards, and live simulators, and train a 350M-parameter world model on it. We identify three distinct hallucination modes: perceptual, action-marginalized, and scene-diverging -- each anchored to a different stage of the pipeline, and develop three signals that accurately predict where the model will fail. To close coverage gaps at training time, we develop a coverage-aware sampling technique; to close them online, our hallucination predictors serve as curiosity rewards for targeted data collection, yielding a data-efficient finetuning recipe that adapts the pretrained world model to entirely unseen environments with as few as 50 real environment trajectories. Overall, our findings reveal that hallucination in world models is inherently a data coverage issue, and that the same signals used to detect it can also be used for mitigation. An interactive web version of our paper is available at https://www.nicklashansen.com/mmbench2" +52dd4accc3b85bdc97a91a5b359271d0a00f556c,2606.26467,10.48550/arXiv.2606.26467,2026,A Causal Foundation Model for Structure and Outcome Prediction,Max Zhu,4,arXiv.org,2,"We introduce TabPFN-CFM, a causal foundation model that can handle multiple causal problems. TabPFN-CFM predicts both causal structure and outcomes from observational data, supports queries on all three levels of Pearl's Causal Hierarchy and uses known graph structure when available to improve predictions. TabPFN-CFM is trained on synthetic datasets, and generalises to real datasets, demonstrating improved performance over both structural and outcome prediction baselines." +662ce403a76826f17aca6bd4d1ba8eaf1184fbea,2606.27345,,2026,SCoPE: Sightline-Coordinate Positional Encoding for Video Diffusion Transformers,Minghao Yin,6,,6,"Video diffusion transformers address their tokens by position on the pixel-time grid: an address in the tensor, not in the world. The address we would want, the world point a token depicts, lies on a surface not yet generated, while its camera ray is fixed once the user specifies a trajectory. SCoPE therefore treats the ray as a second positional coordinate, and camera control becomes a property of the coordinate system, not an added module. The ray is added to the pretrained attention's queries and keys, and the score gains a term that reads the two rays alone. Its canonical form, the reciprocal product of line geometry, measures how nearly two lines of sight meet. Normalize-Gate-Inject makes a single encoding trainable across metric and up-to-scale pose sources. The retrofit keeps RoPE bit-exact, starts from the unchanged pretrained DiT, and adds under 0.1/% new parameters. On Wan2.2 at 5B and 14B under matched data and budget, SCoPE improves every camera-controllability and fidelity metric, leads all closed-loop revisit metrics, and shows widening margins with model size. At 14B, rotation error falls 29/% and FVD 43/% below the strongest baseline." +585fee95a2c62aae3264f6179fc702bec1956cb5,2606.24320,10.48550/arXiv.2606.24320,2026,ZONOS2 Technical Report,Michael G. Clark,5,arXiv.org,0,"We present ZONOS2 8B, our latest TTS model, which achieves state-of-the-art naturalness, prosody, and voice cloning fidelity. We improve upon Zonos-v0.1 across scale, data, and training recipe. We scale the model from 1.6B to 8B total parameters (900M active) with a novel mixture-of-experts (MoE) backbone, improving inference latency and throughput. We expand our training corpus from 200K to over 6M hours using a new data processing pipeline, and we simplify our post-training and conditioning recipes to improve naturalness and voice cloning fidelity. We evaluate ZONOS2 8B on quality, speaker similarity, WER, and ZTTS1-Eval, our novel TTS benchmark, where it performs competitively with state-of-the-art systems while maintaining good streaming latency. We release our model weights and example inference code under an Apache 2.0 license on GitHub and Hugging Face." +e3aa38c070eed8cc1f62f4e4e7416143b36c7899,2607.09694,10.48550/arXiv.2607.09694,2026,Low-Rank Attention Residuals,Jonathan Su,1,arXiv.org,1,"Attention Residuals (AttnRes) replace the fixed residual sum with depth-wise attention over previous sub-layer outputs in Large Language Models (LLMs), but use each output as both a full-dimensional key and value. This couples routing with representation and makes the cost of computing depth-routing scores scale with hidden width $d$. We propose Low-Rank Attention Residuals (LR-AttnRes), which keep full-dimensional residual values while using $r$-dimensional keys, with $r