Masters_Thesis/02-literature/search/screening.csv at 29e323b2fcf3fadaea73563ffd2a28a19e7d6ed9

0 Fork 0

Files

T

Cormac Shannon 95b5857189 Add literature review protocol, searches and first snowball round

Six seeds snowballed (Semantic Scholar) plus logged keyword searches
(OpenAlex, arXiv): 2,127 unique records, 480 title/abstract screened.
AI first-pass suggestions are in their own column; decisions are pending.

Claude-Session: https://claude.ai/code/session_01S8KN7NuUCm8nXLmy6eDuVv

2026-10-28 18:00:06 +00:00

2.9 MiB

Raw Blame History

1	key	year	title	first_author	venue	citations	found_by	n_sources	auto_stage	ai_proposal	abstract
2	arxiv:2511.21377	2025	Controlling changes to attention logits	Ben Anson	arXiv	3	A1-arxiv-qknorm;S1-openalex-qknorm;S2-openalex-logit-instability;snowball-fwd-henry2020;snowball-fwd-stability2024;snowball-fwd-vit22b;snowball-fwd-wortsman2024;snowball-fwd-zhai2023	8	SCREEN	include [IC*] QK-norm vs alternatives for logit control; LR ablations	Stability of neural network weights is critical when training transformer models. The query and key weights are particularly problematic, as they tend to grow large without any intervention. Applying normalization to queries and keys, known as `QK norm', fixes stability issues in practice, but is not always applicable. For example, QK norm is not compatible with Multi Latent Attention (MLA) because QK norm requires full materialization of queries and keys during inference, which is not done in MLA. In this paper we suggest that controlling the changes to logits is important for stability. We show that these changes are controllable by assigning parameter-dependent learning rates to the query and key weights. We find that our cheap intervention allows us to increase the base learning rate of the network, outperform other methods in the MLA setting, and achieve performance competitive with QK norm when using Multi-head Attention.
3	arxiv:2608.02091	2026	One QK Channel, Many Sources: Tracing Low-Precision Attention Collapse	Shuxiao Xie	arXiv	0	A1-arxiv-qknorm;S1-openalex-qknorm;snowball-fwd-henry2020;snowball-fwd-stability2024;snowball-fwd-wortsman2024;snowball-fwd-zhai2023	6	SCREEN	maybe [IM] low-precision attention collapse; QK-norm disrupts it	A bfloat16 transformer can train normally, then collapse abruptly. Prior work links collapse to structured attention errors and shows QK normalization disrupts their compounding. Distinct low-precision errors trigger the same collapse, leaving unclear whether each needs a fix at its source or one shared route can be blocked instead. We isolated the fault behind a reproduced GPT-2-class collapse to the streaming-softmax accumulator, where an fp32 streaming core repairs it, and turned it into an assay for moving a controlled error across sources. Using it, we found that errors placed outside attention still drove the same QK spectral runaway, and that correcting only QK kept training stable while the fault stayed active. This is a source-channel dissociation: fault source is not failure channel. It held across tested architectures and scales, and reproduced on a second GPU architecture. As a causal probe, projecting each update off the current QK weights' leading three singular directions held the query projection's largest singular value to 11.1, whereas removing equal energy elsewhere left it at 237: the QK channel causally drives the early runaway. What lets the injected error in is temporal sign-coherence, its per-head sign persisting across steps, not aggregate deviation; once inside, the runaway shows as attention-logit saturation. QK-Guard, a dormant controller, tests this by switching on parameter-free QK normalization at the first monitored threshold crossing. On the runs designated for this test, the QK-local action prevented the failure of each matched or same-configuration unguarded run; on plain GPT-2, all 12 final train and validation losses were within 0.03 nat of same-configuration always-on QK-norm, and both methods ran 60k steps without collapse. Intervention at the QK locus therefore suffices in place of a fix at each source.
4	title:variancesensitivityinducesattentionentropycollapseintransformers		Variance Sensitivity Induces Attention Entropy Collapse in Transformers	J. Hong		0	snowball-fwd-henry2020;snowball-fwd-stability2024;snowball-fwd-vit22b;snowball-fwd-wortsman2024;snowball-fwd-zhai2023	5	SCREEN	exclude [E4] duplicate record
5	arxiv:2410.16682	2024	Methods of improving LLM training stability	Rybakov, Oleg	arXiv (Cornell University)	16	S3-openalex-small-scale;snowball-fwd-henry2020;snowball-fwd-vit22b;snowball-fwd-wortsman2024;snowball-fwd-zhai2023	5	SCREEN	include [IC*] QK-norm + softcap -> 1.5x LR before divergence, 830M	Training stability of large language models(LLMs) is an important research topic. Reproducing training instabilities can be costly, so we use a small language model with 830M parameters and experiment with higher learning rates to force models to diverge. One of the sources of training instability is the growth of logits in attention layers. We extend the focus of the previous work and look not only at the magnitude of the logits but at all outputs of linear layers in the Transformer block. We observe that with a high learning rate the L2 norm of all linear layer outputs can grow with each training step and the model diverges. Specifically we observe that QKV, Proj and FC2 layers have the largest growth of the output magnitude. This prompts us to explore several options: 1) apply layer normalization not only after QK layers but also after Proj and FC2 layers too; 2) apply layer normalization after the QKV layer (and remove pre normalization). 3) apply QK layer normalization together with softmax capping. We show that with the last two methods we can increase learning rate by 1.5x (without model divergence) in comparison to an approach based on QK layer normalization only. Also we observe significant perplexity improvements for all three methods in comparison to the baseline model.
6	arxiv:2501.18795	2025	Rope to Nope and Back Again: A New Hybrid Attention Strategy	Bowen Yang	arXiv	52	A1-arxiv-qknorm;S1-openalex-qknorm;snowball-fwd-henry2020;snowball-fwd-stability2024;snowball-fwd-vit22b	5	SCREEN	maybe [IC] RoPE/NoPE/QK-norm attention patterns, long context focus	Long-context large language models (LLMs) have achieved remarkable advancements, driven by techniques like Rotary Position Embedding (RoPE) (Su et al., 2023) and its extensions (Chen et al., 2023; Liu et al., 2024c; Peng et al., 2023). By adjusting RoPE parameters and incorporating training data with extended contexts, we can train performant models with considerably longer input sequences. However, existing RoPE-based methods exhibit performance limitations when applied to extended context lengths. This paper presents a comprehensive analysis of various attention mechanisms, including RoPE, No Positional Embedding (NoPE), and Query-Key Normalization (QK-Norm), identifying their strengths and shortcomings in long-context modeling. Our investigation identifies distinctive attention patterns in these methods and highlights their impact on long-context performance, providing valuable insights for architectural design. Building on these findings, we propose a novel architecture featuring a hybrid attention mechanism that integrates global and local attention spans. This design not only surpasses conventional RoPE-based transformer models with full attention in both long and short context tasks but also delivers substantial efficiency gains during training and inference.
7	arxiv:2507.13338	2025	Training Transformers with Enforced Lipschitz Constants	Laker Newhouse	arXiv	19	A1-arxiv-qknorm;S1-openalex-qknorm;snowball-fwd-henry2020;snowball-fwd-vit22b;snowball-fwd-wortsman2024	5	SCREEN	maybe [IA] Lipschitz transformers trained without QK-norm/softcap	Neural networks are often highly sensitive to input and weight perturbations. This sensitivity has been linked to pathologies such as vulnerability to adversarial examples, divergent training, and overfitting. To combat these problems, past research has looked at building neural networks entirely from Lipschitz components. However, these techniques have not matured to the point where researchers have trained a modern architecture such as a transformer with a Lipschitz certificate enforced beyond initialization. To explore this gap, we begin by developing and benchmarking novel, computationally-efficient tools for maintaining norm-constrained weight matrices. Applying these tools, we are able to train transformer models with Lipschitz bounds enforced throughout training. We find that optimizer dynamics matter: switching from AdamW to Muon improves standard methods -- weight decay and spectral normalization -- allowing models to reach equal performance with a lower Lipschitz bound. Inspired by Muon's update having a fixed spectral norm, we co-design a weight constraint method that improves the Lipschitz vs. performance tradeoff on MLPs and 2M parameter transformers. Our 2-Lipschitz transformer on Shakespeare text reaches validation accuracy 60%. Scaling to 145M parameters, our 10-Lipschitz transformer reaches 21% accuracy on internet text. However, to match the NanoGPT baseline validation accuracy of 39.4%, our Lipschitz upper bound increases to 10^264. Nonetheless, our Lipschitz transformers train without stability measures such as layer norm, QK norm, and logit tanh softcapping.
8	arxiv:2604.00726	2026	Exploring Silent Data Corruption as a Reliability Challenge in LLM Training	Anton Altenbernd	arXiv	5	A2-arxiv-logit-instability;S2-openalex-logit-instability;snowball-fwd-vit22b;snowball-fwd-wortsman2024;snowball-fwd-zhai2023	5	SCREEN	exclude [E5] title/abstract: not about attention-logit control or LM training stability	As Large Language Models (LLMs) scale in size and complexity, the consequences of failures during training become increasingly severe. A major challenge arises from Silent Data Corruption (SDC): hardware-induced faults that bypass system-level detection mechanisms. SDC may behave like benign numerical noise, but can also cause harmful gradient corruption that leads to loss spikes, divergence, or stalled progress. This work provides a controlled study of how intermittent SDC affects LLM pretraining. Using targeted fault injection at the level of GPU matrix-multiply instructions, we characterize the sensitivity of different bit positions, kernel functions, and execution stages. Our analysis shows that locally originating faults can produce impactful corruption, including NaN propagation, short-lived spikes in loss, gradient norm, and attention logits, as well as persistent parameter divergence. Building on the observed corruption signatures, we propose a lightweight detection method that identifies potentially harmful parameter updates. Experiments on LLaMA models with 60M, 350M, and 1.3B parameters demonstrate that recomputing the most recent training step upon detection can effectively mitigate the impact of these events.
9	arxiv:2303.06296	2023	Stabilizing Transformer Training by Preventing Attention Entropy Collapse	Shuangfei Zhai	arXiv	221	A2-arxiv-logit-instability;S2-openalex-logit-instability;snowball-back-stability2024;snowball-back-wortsman2024	4	SCREEN	include [SEED] attention entropy collapse <-> instability	Training stability is of great importance to Transformers. In this work, we investigate the training dynamics of Transformers by examining the evolution of the attention layers. In particular, we track the attention entropy for each attention head during the course of training, which is a proxy for model sharpness. We identify a common pattern across different architectures and tasks, where low attention entropy is accompanied by high training instability, which can take the form of oscillating loss or divergence. We denote the pathologically low attention entropy, corresponding to highly concentrated attention scores, as $\textit{entropy collapse}$. As a remedy, we propose $σ$Reparam, a simple and efficient solution where we reparametrize all linear layers with spectral normalization and an additional learned scalar. We demonstrate that $σ$Reparam successfully prevents entropy collapse in the attention layers, promoting more stable training. Additionally, we prove a tight lower bound of the attention entropy, which decreases exponentially fast with the spectral norm of the attention logits, providing additional motivation for our approach. We conduct experiments with $σ$Reparam on image classification, image self-supervised learning, machine translation, speech recognition, and language modeling tasks. We show that $σ$Reparam provides stability and robustness with respect to the choice of hyperparameters, going so far as enabling training (a) a Vision Transformer {to competitive performance} without warmup, weight decay, layer normalization or adaptive optimizers; (b) deep architectures in machine translation and (c) speech recognition to competitive performance without warmup and adaptive optimizers. Code is available at \url{https://github.com/apple/ml-sigma-reparam}.
10	arxiv:2304.13013	2023	Stable and low-precision training for large-scale vision-language models	Mitchell Wortsman	Neural Information Processing Systems	106	snowball-back-stability2024;snowball-back-wortsman2024;snowball-fwd-vit22b;snowball-fwd-zhai2023	4	SCREEN	include [IM] Wortsman et al. follow-up on instability	We introduce new methods for 1) accelerating and 2) stabilizing training for large language-vision models. 1) For acceleration, we introduce SwitchBack, a linear layer for int8 quantized training which provides a speed-up of 13-25% while matching the performance of bfloat16 training within 0.1 percentage points for the 1B parameter CLIP ViT-Huge -- the largest int8 training to date. Our main focus is int8 as GPU support for float8 is rare, though we also analyze float8 training through simulation. While SwitchBack proves effective for float8, we show that standard techniques are also successful if the network is trained and initialized so that large feature magnitudes are discouraged, which we accomplish via layer-scale initialized with zeros. 2) For stability, we analyze loss spikes and find they consistently occur 1-8 iterations after the squared gradients become under-estimated by their AdamW second moment estimator. As a result, we recommend an AdamW-Adafactor hybrid which avoids loss spikes when training a CLIP ViT-Huge model and outperforms gradient clipping at the scales we test.
11	arxiv:2410.07563	2024	PLaMo-100B: A Ground-Up Language Model Designed for Japanese Proficiency	Preferred Elements	arXiv	0	A1-arxiv-qknorm;S1-openalex-qknorm;snowball-fwd-henry2020;snowball-fwd-wortsman2024	4	SCREEN	exclude [E5] title/abstract: not about attention-logit control or LM training stability	We introduce PLaMo-100B, a large-scale language model designed for Japanese proficiency. The model was trained from scratch using 2 trillion tokens, with architecture such as QK Normalization and Z-Loss to ensure training stability during the training process. Post-training techniques, including Supervised Fine-Tuning and Direct Preference Optimization, were applied to refine the model's performance. Benchmark evaluations suggest that PLaMo-100B performs well, particularly in Japanese-specific tasks, achieving results that are competitive with frontier models like GPT-4. The base model is available at https://huggingface.co/pfnet/plamo-100b.
12	arxiv:2406.17837	2024	Transformer Normalisation Layers and the Independence of Semantic Subspaces	Stephen Menary	arXiv.org	7	snowball-back-stability2024;snowball-fwd-henry2020;snowball-fwd-vit22b;snowball-fwd-wortsman2024	4	SCREEN	exclude [E5] title/abstract: not about attention-logit control or LM training stability	Recent works have shown that transformers can solve contextual reasoning tasks by internally executing computational graphs called circuits. Circuits often use attention to logically match information from subspaces of the representation, e.g. using position-in-sequence to identify the previous token. In this work, we consider a semantic subspace to be any independent subspace of the latent representation that can fully determine an attention distribution. We show that Pre-Norm, the placement of normalisation layer used by state-of-the-art transformers, violates this ability unless the model learns a strict representation structure of orthogonal spheres. This is because it causes linear subspaces to interfere through their common normalisation factor. Theoretically, we analyse circuit stability by modelling this interference as random noise on the $L_2$-norms of the query/key/value vectors, predicting a phenomenon of circuit collapse when sparse-attention shifts to a different token. Empirically, we investigate the sensitivity of real-world models trained for mathematical addition, observing a 1% rate of circuit collapse when the norms are artificially perturbed by $\lesssim$10%. We contrast Pre-Norm with QKV-Norm, which places normalisation after the attention head's linear operators. Theoretically this relaxes the representational constraints. Empirically we observe comparable in-distribution but worse out-of-distribution performance.
13	arxiv:2410.13060	2024	AERO: Entropy-Guided Framework for Private LLM Inference	Nandan Kumar Jha		6	snowball-fwd-stability2024;snowball-fwd-vit22b;snowball-fwd-wortsman2024;snowball-fwd-zhai2023	4	SCREEN	exclude [E5] title/abstract: not about attention-logit control or LM training stability	Privacy-preserving computation enables language model inference directly on encrypted data yet suffers from prohibitive latency and communication overheads, primarily due to nonlinear functions. Removing nonlinearities, however, can trigger one of two failure modes restricting the potential for nonlinearity removal: entropy collapse in deeper layers, which destabilizes training, and entropic overload in early layers, causing under-utilization of attention heads. To address these challenges, we introduce AERO, an entropy-guided framework to strategically eliminates costly nonlinear operations from transformer architectures, which employs an adaptive recalibration through a head-wise entropy regularizer with learnable per-head strengths, enabling each head to adjust its entropy level while penalizing extreme entropies and fostering functional diversity through a tolerance margin. Experiments show AERO can save 3.4$\times$ communication and 1.4$\times$ latency, without any performance penalty.
14	arxiv:2505.22014	2025	Learning in Compact Spaces with Approximately Normalized Transformer	Jörg K. H. Franke	arXiv	5	A1-arxiv-qknorm;S1-openalex-qknorm;snowball-fwd-henry2020;snowball-fwd-wortsman2024	4	SCREEN	maybe [IA] approx. normalised transformer vs GPT+QK-norm convergence	The successful training of deep neural networks requires addressing challenges such as overfitting, numerical instabilities leading to divergence, and increasing variance in the residual stream. A common solution is to apply regularization and normalization techniques that usually require tuning additional hyperparameters. An alternative is to force all parameters and representations to lie on a hypersphere. This removes the need for regularization and increases convergence speed, but comes with additional costs. In this work, we propose a more holistic, approximate normalization via simple scalar multiplications motivated by the tight concentration of the norms of high-dimensional random vectors. Additionally, instead of applying strict normalization for the parameters, we constrain their norms. These modifications remove the need for weight decay and learning rate warm-up as well, but do not increase the total number of normalization layers. Our experiments with transformer architectures show up to 40% faster convergence compared to GPT models with QK normalization, with only 3% additional runtime cost. When deriving scaling laws, we found that our method enables training with larger batch sizes while preserving the favorable scaling characteristics of classic GPT architectures.
15	arxiv:2501.12352	2025	Test-time regression: a unifying framework for designing sequence models with associative memory	Ke Alexander Wang	arXiv	78	A1-arxiv-qknorm;S1-openalex-qknorm;snowball-fwd-vit22b;snowball-fwd-wortsman2024	4	SCREEN	maybe [IM] theory justifying QK-norm (test-time regression view)	Sequence models lie at the heart of modern deep learning. However, rapid advancements have produced a diversity of seemingly unrelated architectures, such as Transformers and recurrent alternatives. In this paper, we introduce a unifying framework to understand and derive these sequence models, inspired by the empirical importance of associative recall, the capability to retrieve contextually relevant tokens. We formalize associative recall as a two-step process, memorization and retrieval, casting memorization as a regression problem. Layers that combine these two steps perform associative recall via ``test-time regression'' over its input tokens. Prominent layers, including linear attention, state-space models, fast-weight programmers, online learners, and softmax attention, arise as special cases defined by three design choices: the regression weights, the regressor function class, and the test-time optimization algorithm. Our approach clarifies how linear attention fails to capture inter-token correlations and offers a mathematical justification for the empirical effectiveness of query-key normalization in softmax attention. Further, it illuminates unexplored regions within the design space, which we use to derive novel higher-order generalizations of softmax attention. Beyond unification, our work bridges sequence modeling with classic regression methods, a field with extensive literature, paving the way for developing more powerful and theoretically principled architectures.
16	arxiv:2505.15548	2025	Dense Local Dependencies Induce Attention-Logit Explosion and Training Instability During Long-Sequence Transformer Training	Suvadeep Hajra	arXiv	2	A2-arxiv-logit-instability;S2-openalex-logit-instability;snowball-fwd-henry2020;snowball-fwd-wortsman2024	4	SCREEN	include [IM] dense local dependencies -> attention-logit explosion	Autoregressive transformer language models frequently exhibit training instability when trained on long sequences, particularly under low-precision arithmetic. Although this instability is often accompanied by attention-logit explosion, its underlying cause remains poorly understood. In this work, we present analytical insights and empirical evidence that dense local dependencies are a major contributor to attention-logit explosion. We demonstrate that dense local dependency patterns yield an effectively high-rank attention structure, which the low-rank parameterization of self-attention can only approximate with increasingly large logits as the sequence length grows. This logit inflation ultimately leads to training instability under low-precision arithmetic. We support this explanation through extensive experiments on synthetic and language modeling tasks. Our results consistently show that attention-logit growth increases with sequence length, is mitigated by increasing the attention dimension, and is substantially reduced by explicitly modeling dense local dependencies. Furthermore, we show that this growth is driven by the density of local dependencies rather than by locality alone. More broadly, our findings suggest that explicitly modeling dense local dependencies constitutes an important design principle for developing stable, efficient, and scalable long-context transformer architectures for autoregressive language modeling.
17	arxiv:2505.21910	2025	Taming Transformer Without Using Learning Rate Warmup	Xianbiao Qi	arXiv	9	A2-arxiv-logit-instability;snowball-fwd-henry2020;snowball-fwd-vit22b;snowball-fwd-zhai2023	4	SCREEN	include [IM] W_q^T W_k spectral concentration -> entropy collapse, warmup	Scaling Transformer to a large scale without using some technical tricks such as learning rate warump and using an obviously lower learning rate is an extremely challenging task, and is increasingly gaining more attention. In this paper, we provide a theoretical analysis for the process of training Transformer and reveal the rationale behind the model crash phenomenon in the training process, termed \textit{spectral energy concentration} of ${\bW_q}^{\top} \bW_k$, which is the reason for a malignant entropy collapse, where ${\bW_q}$ and $\bW_k$ are the projection matrices for the query and the key in Transformer, respectively. To remedy this problem, motivated by \textit{Weyl's Inequality}, we present a novel optimization strategy, \ie, making the weight updating in successive steps smooth -- if the ratio $\frac{σ_{1}(\nabla \bW_t)}{σ_{1}(\bW_{t-1})}$ is larger than a threshold, we will automatically bound the learning rate to a weighted multiple of $\frac{σ_{1}(\bW_{t-1})}{σ_{1}(\nabla \bW_t)}$, where $\nabla \bW_t$ is the updating quantity in step $t$. Such an optimization strategy can prevent spectral energy concentration to only a few directions, and thus can avoid malignant entropy collapse which will trigger the model crash. We conduct extensive experiments using ViT, Swin-Transformer and GPT, showing that our optimization strategy can effectively and stably train these Transformers without using learning rate warmup.
18	arxiv:2606.16310	2026	QK-Normed MLA: QK normalization without full key caching	Yizhou Han	arXiv	1	A1-arxiv-qknorm;S1-openalex-qknorm;snowball-fwd-henry2020;snowball-fwd-vit22b	4	SCREEN	include [IC] QK-norm permits higher LR; high-LR stress test vs QK-clip	Query-key (QK) normalization stabilizes attention by controlling the scale of queries and keys before the dot product, but is not immediately compatible with Multi-head Latent Attention (MLA). MLA achieves efficient decoding by caching low-dimensional latent states instead of full keys, whereas post-projection QK RMSNorm appears to require the fully projected key for every cached token. We show this apparent incompatibility is an implementation artifact, not an architectural constraint. RMSNorm decomposes into a static affine weight and a dynamic scalar RMS statistic. The static key-side weight can be absorbed into the MLA query-side projection; the dynamic key statistic reduces to one inverse-RMS scalar per token and KV group. The resulting formulation is exactly equivalent to explicit post-projection QK RMSNorm in exact arithmetic and preserves MLA's latent decode path. In our 400M runs trained for up to 100B tokens, QK-Normed MLA achieves lower training loss and better downstream accuracy than QK clipping, while H800 decode benchmarks show less than 2% latency overhead up to 256k context. These results make QK normalization a practical stabilization option for MLA models without requiring full-key caching.
19	arxiv:2602.02522	2026	IMU-1: Sample-Efficient Pre-training of Small Language Models	George Grigorev	arXiv	1	A1-arxiv-qknorm;S1-openalex-qknorm;snowball-fwd-henry2020;snowball-fwd-vit22b	4	SCREEN	include [SEED*] QK-norm largest win at 70M, bundled with muP	We present IMU-1, a 430M-parameter language model trained on 72B tokens that approaches the benchmark performance of models trained on 56x more data. We describe a validated training recipe combining recent architectural interventions (QK-norm attention, per-head gating, value residuals, LayerNorm scaling) with optimization advances (NorMuon with cautious weight decay, muP parametrization) and a three-stage training schedule with post-hoc checkpoint EMA. We provide ablations for each component and release code, weights and data to enable reproduction: https://huggingface.co/thepowerfuldeez/imu1_base
20	arxiv:2603.02170	2026	SageBwd: A Trainable Low-bit Attention	Jintao Zhang	arXiv	10	A1-arxiv-qknorm;S1-openalex-qknorm;snowball-fwd-henry2020;snowball-fwd-vit22b	4	SCREEN	include [IC] QK-norm necessary for stable low-bit training at large tokens/step	Low-bit attention, such as SageAttention, has emerged as an effective approach for accelerating model inference, but its applicability to training remains poorly understood. In prior work, we introduced SageBwd, a trainable INT8 attention that quantizes six of seven attention matrix multiplications while preserving fine-tuning performance. However, SageBwd exhibited a persistent performance gap to full-precision attention (FPA) during pre-training. In this work, we investigate why this gap occurs and demonstrate that SageBwd matches full-precision attention during pretraining. Through experiments and theoretical analysis, we reach a few important insights and conclusions: (i) QK-norm is necessary for stable training at large tokens per step, (ii) quantization errors primarily arise from the backward-pass score gradient dS, (iii) reducing tokens per step enables SageBwd to match FPA performance in pre-training, and (iv) K-smoothing remains essential for training stability, while Q-smoothing provides limited benefit during pre-training.
21	arxiv:2607.18363	2026	A Controlled Study of Attention-Only Transformers	Henry Ndubuaku	arXiv	1	A1-arxiv-qknorm;S1-openalex-qknorm;snowball-fwd-henry2020;snowball-fwd-vit22b	4	SCREEN	include [IC*] QK-norm keeps 48-layer stacks trainable: depth evidence (H3)	Feed-forward networks hold two thirds of a transformer's non-embedding parameters, yet the architecture has not received a necessity test that controls parameters, compute, and depth at once. We pretrain attention-only decoder transformers (Simple Attention Networks, SANs) against standard transformers matched separately for parameter count, training FLOPs, and depth (2 to 48 layers), for up to 105B tokens at 6M to 87M parameters. Deleting feed-forward layers in place is costly: the standard transformer leads by 0.47 nats at matched depth and 0.26 nats at matched FLOPs. Reallocating the freed budget into attention depth closes the gap: at matched parameters the difference is 0.006 nats (0.27 percent of loss), reproducible to one part in ten thousand across seed pairs, shrinking across 5B, 30B, and 105B budgets, and holding near 0.02 nats across a 29x size range. Three measurements localize the remaining gap to parametric recall: attention-only models are better on context-grounded answers and worse where knowledge must come from weights. Weight spectra show why: routing matrices (Q/K) crystallize early, content matrices accumulate rank slowly, and removing feed-forward layers relocates this accumulation to the attention output projection. QK-normalization, not feed-forward layers or residual gating, keeps 48-layer attention-only stacks trainable. The deficit concentrates on low-context query prediction and localizes there entirely by the largest budget. A pre-registered test confirms the account: it predicts a 0.02 to 0.05 nat gap on knowledge-dense web text; a matched pair trained on fineweb-edu measures 0.040. Within the tested regime, attention does the rest.
22	arxiv:2610.34582	2026	HobbyLM: A Sparse Mixture-of-Experts Model Family on a Hobby Budget	Harish Santhanalakshmi Ganesan	arXiv	0	A1-arxiv-qknorm;S1-openalex-qknorm;snowball-fwd-henry2020;snowball-fwd-vit22b	4	SCREEN	include [IS*] single-variable ablations at 130M incl. QK-norm (HobbyLM)	We describe HobbyLM, a family of small sparse Mixture-of-Experts (MoE) models trained from scratch with a few hundred to a few thousand dollars of rented GPU time. The core is a decoder-only transformer with fine-grained experts, sigmoid routing with auxiliary-loss-free load balancing, grouped-query attention with per-head QK-normalization, and aMuon/AdamW optimizer split. Each of these choices was selected with a single-variable ablation suite at 130M parameters. On this architecture we train a 500M-parameter model (169M active) on 40B FineWeb tokens and post-train it into a chat model, a masked-diffusion language model, a function-calling and computer-use agent, and a single "omni" checkpoint that reads text, images, video, sound and speech through frozen encoders and small projectors. We run these models on a laptop CPU with a from-scratch Rust inference engine. Two of the authors then adopted the architecture without architectural changes to build HobbyLM-1B (1.04B total, 305M active parameters), trained on a 100B-token curated mixture with a high-quality anneal. HobbyLM-1B reaches a 47.5 average on seven zero-shot benchmarks, and its builders also release an instruction-tuned model and an on-device macOS computer-use adapter. Finally, we describe HobbyLM-Image, a 333M-parameter flow-matching diffusion transformer that generates 1024x1024 images in a 32x-compressed latent space, together with a dependency-light CPU implementation of the full text-to-image pipeline. We report what worked, what did not, and where the small-scale ceilings lie. We also state for each component which author designed or built it.
23	arxiv:2610.22330	2026	Designing Transformers for Continuous Diffusion Language Models	Weijian Luo	arXiv	0	A1-arxiv-qknorm;S1-openalex-qknorm;snowball-fwd-henry2020;snowball-fwd-zhai2023	4	SCREEN	exclude [E5] title/abstract: not about attention-logit control or LM training stability	Continuous diffusion language models (CDLMs) offer an alternative approach to parallel language generation; however, most existing studies directly use transformer architectures from either causal large language models (LLMs) or Diffusion Transformers (DiTs) for image generation, which were not optimized for CDLMs. In this paper, we conduct a systematic study on Transformer architectures for CDLMs and introduce Language Diffusion Transformer (LangDiT), combining feature and query-key normalization, wider noise conditioning, likelihood-shortcut gradient routing and calibration, and confidence-gated training self-conditioning. LangDiT significantly reduces generation perplexity in matched LangFlow comparisons, establishing a new state of the art in continuous language generation across different model formulations. Using LangFlow as a representative CDLM, LangDiT as a plug-and-play backbone leads to a new state-of-the-art model with a generation perplexity (GenPPL) of 21.72 using 1024 NFEs and 44.72 GenPPL using 128 NFEs on the unconditional generation of 1024-token sequences on the OpenWebText dataset. When adapted to Plaid-style CDLM formulations, LangDiT results in a new leading GenPPL of 11.33 using 1024 denoising NFEs, and 27.51 GenPPL using 128 denoising NFEs. These gains are achieved while retaining each model's underlying representation, training objective, and inference sampler, isolating transformer architecture as a fundamental source of improvement. Our study identifies architectural design as an important source of improvement in continuous diffusion language models.
24	arxiv:2604.01683	2026	Invertible Query-Key Coupling Composes with Attention Mechanisms	Barak Gahtan	arXiv	0	A1-arxiv-qknorm;S1-openalex-qknorm;snowball-fwd-henry2020;snowball-fwd-zhai2023	4	SCREEN	exclude [E5] title/abstract: not about attention-logit control or LM training stability	Scaled dot-product attention forms its queries and keys as independent linear projections, so the two never interact before the dot product that scores them. We study coupled query-key dynamics, a pre-scoring transformation that evolves each token's query and key jointly through a shared invertible coupling before standard scoring. We realize it as an alternating affine map in the style of real non-volume-preserving flows: the coupling is the identity at initialization, adds a small fraction of parameters per head, and leaves the softmax and surrounding architecture unchanged. We place coupling on top of existing attention methods rather than replacing them, and ask whether that composition helps. On WikiText-103, adding coupling to Differential Attention improves on it at both 150M and 455M parameters. At 455M the gain is significant at sequence length 512 (p=0.003, six seeds), survives a Bonferroni correction and replicates on a held-out test split; it also holds across rotary-embedding training lengths 512, 1024, and 2048. The same additive direction appears when coupling is added to query-key normalization (significant at 150M) and Multi-Token Attention (directional). Matched controls attribute the gain to the joint pre-scoring coupling rather than to added capacity, and show that removing the invertibility guarantee preserves the 455M gain yet is far worse than the base method at 150M, so invertibility is what makes the coupling reliable across scales. On its own, coupling lowers perplexity at 60M and 150M (one-sided Welch tests, p<0.05) but the gain narrows with scale and is not significant at 455M. We relate the construction to the expressivity of coupling flows, use an associative-recall study to map where coupling helps and where it degrades sharp retrieval, and conclude that coupling is most useful in composition with a scoring-stage method rather than as a standalone change.
25	arxiv:2609.11655	2026	Musec: MomentUm SpEctral Clipping for Stable Muon-type Training	Zhuanghua Liu	arXiv	2	A2-arxiv-logit-instability;S2-openalex-logit-instability;snowball-fwd-vit22b;snowball-fwd-wortsman2024	4	SCREEN	exclude [E5] title/abstract: not about attention-logit control or LM training stability	Muon has emerged as a highly effective optimizer for large language model training, often achieving superior convergence and performance compared with the widely adopted Adam and AdamW optimizers. Nevertheless, Muon is prone to training instability due to its spectral flattening, manifested by loss spikes and unbounded growth of model weights. Existing approaches primarily rely on weight or attention-logit clipping, which require architecture-specific modifications and do not directly address instability across all model components. We propose MomentUm SpEctral Clipping (Musec), which replaces Muon's spectral flattening with spectral clipping: rather than setting all singular values of the momentum matrix to approximately one, Musec clips singular values that exceed a threshold while preserving the underlying spectral structure of the momentum. Our strategy provides an optimizer-level, architecture-agnostic mechanism for stabilizing Muon training. Theoretically, we establish convergence guarantees for Musec in nonconvex nonsmooth stochastic optimization. Practically, we develop Soft Musec, an efficient implementation that uses a smooth spectral saturation function approximated by coupled Newton-Schulz iterations. Empirically, Soft Musec consistently improves training stability over existing Muon variants across a wide range of learning rates and model sizes, remaining stable in settings where existing Muon variants diverge while matching their performance under well-tuned configurations.
26	arxiv:2609.34272	2026	Broken Symmetry in BF16 Attention: Why FlashAttention Gradients Blow Up Late in Training	Jun-Lin Chen		0	snowball-fwd-henry2020;snowball-fwd-vit22b;snowball-fwd-wortsman2024;snowball-fwd-zhai2023	4	SCREEN	maybe [IM] BF16 attention gradients; logit growth late in training	BF16 is now standard in large-scale pretraining, including in fused attention kernels such as FlashAttention, and these kernels are widely trusted. When we used FlashAttention-3 to pretrain a 450M-parameter transformer on 50B tokens, however, we ran into a problem: training was healthy for 25B tokens, then the gradient norm grew a thousandfold and the loss ended 0.2 nats above FP32 attention, without a single NaN. Recomputing the attention backward of just two layers in FP32 removes almost all of the excess gradient. Part of the cause is known: a fused multiply-add in the forward softmax, so far treated as an extreme-input NaN case and never fixed in FlashAttention-3. Repairing it stops the blow-up, but the query gradient is still wrong by more than its own size, and training still drives attention logits to thousands of times their size under accurate gradients. The remaining error comes from a broken conservation law. The softmax score gradient sums to zero along every row, which makes the query gradient blind to where the keys sit as a group; rounding it to BF16 leaves a small nonzero sum that leaks the mean key into the gradient, and the leak grows exactly as late training makes keys large and attention sharp. We introduce GProj (gauge projection), which restores the zero sum after the cast with two rank-one corrections per row. It cuts the remaining median query/key gradient errors from 219%/13% to 0.34%/0.37%, on par with FP32 attention, for 4.7% more time per training step. In matched from-scratch runs it trains to the same loss as FP32 attention, while FlashAttention-3 and key smoothing both destabilize.
27	arxiv:2606.16768	2026	Taming Curvature: Architecture Warm-Up for Stable Transformer Training	Sameera Ramasinghe	arXiv.org	1	snowball-fwd-henry2020;snowball-fwd-vit22b;snowball-fwd-wortsman2024;snowball-fwd-zhai2023	4	SCREEN	exclude [E5] title/abstract: not about attention-logit control or LM training stability	Training billion-parameter Transformers is often brittle, with transient loss spikes and divergence that waste compute. Even though the recently developed Edge of Stability (EoS) theory provides a powerful tool to understand and control the stability of optimization methods via the (preconditioned) curvature, these curvature-controlling methods are not popular in large-scale Transformer training due to the complexity of curvature estimation. To this end, we first introduce a fast online estimator of the largest (preconditioned) Hessian eigenvalue (i.e., curvature) based on a warm-started variant for power iteration with Hessian-vector products. We show theoretically, and verify empirically, that the proposed method makes per-iteration curvature tracking feasible at billion parameter scale while being more accurate. Using this tool, we find that training instabilities coincide with surges in preconditioned curvature and that curvature grows with depth. Motivated by these observations, we propose architecture warm-up: progressively growing network depth to carefully control the preconditioned Hessian and stabilize training. Experiments on large Transformers validate that our approach enables efficient curvature tracking and reduces instabilities compared to existing state-of-the-art stabilization techniques without slowing down convergence.
28	doi:10.18653/v1/2020.findings-emnlp.379	2020	Query-Key Normalization for Transformers	Alex Henry	Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP)	45	S1-openalex-qknorm;snowball-back-imu1;snowball-back-stability2024	3	SCREEN	include [SEED] origin of QK-norm (ACL anthology copy of 2010.04245)	Low-resource language translation is a challenging but socially valuable NLP task.Building on recent work adapting the Transformer's normalization to this setting, we propose QKNORM, a normalization technique that modifies the attention mechanism to make the softmax function less prone to arbitrary saturation without sacrificing expressivity.Specifically, we apply ℓ 2 normalization along the head dimension of each query and key matrix prior to multiplying them and then scale up by a learnable parameter instead of dividing by the square root of the embedding dimension.We show improvements averaging 0.928 BLEU over state-of-the-art bilingual benchmarks for 5 low-resource translation pairs from the TED Talks corpus and IWSLT'15. 1
29	arxiv:2302.05442	2023	Scaling Vision Transformers to 22 Billion Parameters	Mostafa Dehghani	International Conference on Machine Learning	972	snowball-back-imu1;snowball-back-stability2024;snowball-back-wortsman2024	3	SCREEN	include [SEED] QK-norm popularised to stop logit growth at 22B	The scaling of Transformers has driven breakthrough capabilities for language models. At present, the largest large language models (LLMs) contain upwards of 100B parameters. Vision Transformers (ViT) have introduced the same architecture to image and video modelling, but these have not yet been successfully scaled to nearly the same degree; the largest dense ViT contains 4B parameters (Chen et al., 2022). We present a recipe for highly efficient and stable training of a 22B-parameter ViT (ViT-22B) and perform a wide variety of experiments on the resulting model. When evaluated on downstream tasks (often with a lightweight linear model on frozen features), ViT-22B demonstrates increasing performance with scale. We further observe other interesting benefits of scale, including an improved tradeoff between fairness and performance, state-of-the-art alignment to human visual perception in terms of shape/texture bias, and improved robustness. ViT-22B demonstrates the potential for"LLM-like"scaling in vision, and provides key steps towards getting there.
30	arxiv:2309.14322	2023	Small-scale proxies for large-scale Transformer training instabilities	Mitchell Wortsman	International Conference on Learning Representations	210	snowball-back-stability2024;snowball-fwd-vit22b;snowball-fwd-zhai2023	3	SCREEN	include [SEED*] LR-sensitivity metric; qk-layernorm at small scale	Teams that have trained large Transformer-based models have reported training instabilities at large scale that did not appear when training with the same hyperparameters at smaller scales. Although the causes of such instabilities are of scientific interest, the amount of resources required to reproduce them has made investigation difficult. In this work, we seek ways to reproduce and study training stability and instability at smaller scales. First, we focus on two sources of training instability described in previous work: the growth of logits in attention layers (Dehghani et al., 2023) and divergence of the output logits from the log probabilities (Chowdhery et al., 2022). By measuring the relationship between learning rate and loss across scales, we show that these instabilities also appear in small models when training at high learning rates, and that mitigations previously employed at large scales are equally effective in this regime. This prompts us to investigate the extent to which other known optimizer and model interventions influence the sensitivity of the final loss to changes in the learning rate. To this end, we study methods such as warm-up, weight decay, and the $\mu$Param (Yang et al., 2022), and combine techniques to train small models that achieve similar losses across orders of magnitude of learning rate variation. Finally, to conclude our exploration we study two cases where instabilities can be predicted before they emerge by examining the scaling behavior of model activation and gradient norms.
31	arxiv:2410.10781	2024	When Attention Sink Emerges in Language Models: An Empirical View	Xiangming Gu	International Conference on Learning Representations	237	snowball-fwd-vit22b;snowball-fwd-wortsman2024;snowball-fwd-zhai2023	3	SCREEN	include [IM] when attention sinks emerge (seed paper ref)	Language Models (LMs) assign significant attention to the first token, even if it is not semantically important, which is known as attention sink. This phenomenon has been widely adopted in applications such as streaming/long context generation, KV cache optimization, inference acceleration, model quantization, and others. Despite its widespread use, a deep understanding of attention sink in LMs is still lacking. In this work, we first demonstrate that attention sinks exist universally in LMs with various inputs, even in small models. Furthermore, attention sink is observed to emerge during the LM pre-training, motivating us to investigate how optimization, data distribution, loss function, and model architecture in LM pre-training influence its emergence. We highlight that attention sink emerges after effective optimization on sufficient training data. The sink position is highly correlated with the loss function and data distribution. Most importantly, we find that attention sink acts more like key biases, storing extra attention scores, which could be non-informative and not contribute to the value computation. We also observe that this phenomenon (at least partially) stems from tokens' inner dependence on attention scores as a result of softmax normalization. After relaxing such dependence by replacing softmax attention with other attention operations, such as sigmoid attention without normalization, attention sinks do not emerge in LMs up to 1B parameters. The code is available at https://github.com/sail-sg/Attention-Sink.
32	arxiv:2402.10202	2024	Bridging Associative Memory and Probabilistic Modeling	Rylan Schaeffer	arXiv.org	7	snowball-fwd-vit22b;snowball-fwd-wortsman2024;snowball-fwd-zhai2023	3	SCREEN	exclude [E5] title/abstract: not about attention-logit control or LM training stability	Associative memory and probabilistic modeling are two fundamental topics in artificial intelligence. The first studies recurrent neural networks designed to denoise, complete and retrieve data, whereas the second studies learning and sampling from probability distributions. Based on the observation that associative memory's energy functions can be seen as probabilistic modeling's negative log likelihoods, we build a bridge between the two that enables useful flow of ideas in both directions. We showcase four examples: First, we propose new energy-based models that flexibly adapt their energy functions to new in-context datasets, an approach we term \textit{in-context learning of energy functions}. Second, we propose two new associative memory models: one that dynamically creates new memories as necessitated by the training data using Bayesian nonparametrics, and another that explicitly computes proportional memory assignments using the evidence lower bound. Third, using tools from associative memory, we analytically and numerically characterize the memory capacity of Gaussian kernel density estimators, a widespread tool in probababilistic modeling. Fourth, we study a widespread implementation choice in transformers -- normalization followed by self attention -- to show it performs clustering on the hypersphere. Altogether, this work urges further exchange of useful ideas between these two continents of artificial intelligence.
33	arxiv:2502.17092	2025	Shakti-VLMs: Scalable Vision-Language Models for Enterprise AI	Syed Abdul Gaffar Shakhadri	arXiv	0	A1-arxiv-qknorm;S1-openalex-qknorm;snowball-fwd-henry2020	3	SCREEN	exclude [E5] title/abstract: not about attention-logit control or LM training stability	We introduce Shakti VLM, a family of vision-language models in the capacity of 1B and 4B parameters designed to address data efficiency challenges in multimodal learning. While recent VLMs achieve strong performance through extensive training data, Shakti models leverage architectural innovations to attain competitive results with fewer tokens. Key advancements include QK-Normalization for attention stability, hybrid normalization techniques, and enhanced positional encoding. A three-stage training strategy further optimizes learning efficiency. Evaluations show that Shakti-Shakti-VLM-1B and Shakti-VLM-4B excel in document understanding, Visual Reasoning, OCR extraction, and general multimodal reasoning. Our results highlight that high performance can be achieved through model design and training strategy rather than sheer data volume, making Shakti an efficient solution for enterprise-scale multimodal tasks.
34	arxiv:2502.00919	2025	Attention Sinks: A 'Catch, Tag, Release' Mechanism for Embeddings	Stephen Zhang	arXiv	18	A1-arxiv-qknorm;S1-openalex-qknorm;snowball-fwd-henry2020	3	SCREEN	maybe [IM] attention sinks persist with QK-norm	Large language models (LLMs) often concentrate their attention on a few specific tokens referred to as attention sinks. Common examples include the first token, a prompt-independent sink, and punctuation tokens, which are prompt-dependent. While the tokens causing the sinks often lack direct semantic meaning, the presence of the sinks is critical for model performance, particularly under model compression and KV-caching. Despite their ubiquity, the function, semantic role, and origin of attention sinks -- especially those beyond the first token -- remain poorly understood. In this work, we conduct a comprehensive investigation demonstrating that attention sinks: catch a sequence of tokens, tag them using a common direction in embedding space, and release them back into the residual stream, where tokens are later retrieved based on the tags they have acquired. Probing experiments reveal these tags carry semantically meaningful information, such as the truth of a statement. These findings extend to reasoning models, where the mechanism spans more heads and explains greater variance in embeddings, or recent models with query-key normalization, where sinks remain just as prevalent. To encourage future theoretical analysis, we introduce a minimal problem which can be solved through the 'catch, tag, release' mechanism, and where it emerges through training.
35	arxiv:2508.20577	2025	MERIT: Maximum-normalized Element-wise Ratio for Language Model Large-batch Training	Yang Luo	arXiv (Cornell University)	0	S2-openalex-logit-instability;snowball-fwd-wortsman2024;snowball-fwd-zhai2023	3	SCREEN	exclude [E5] title/abstract: not about attention-logit control or LM training stability	Large-batch training has become a cornerstone in accelerating the training of deep neural networks, yet it poses challenges in optimization and generalization. Existing optimizers like AdamW present performance degradation during language models' large-batch training, due to the information bottleneck in attention layers caused by the sharp increase of max attention logit. While the LAMB optimizer partially addresses this issue, some attention layers still face this issue. The reason is that $l_2$-norm-based trust ratios in LAMB are less effective in directly influencing the max value of query/key weights. Furthermore, the weight-wise trust ratio in LAMB is error-prone as it overlooks relationships of weight values within rows or columns. Building on these observations, we propose a novel optimizer, MERIT, which leverages the max-norm to calculate the trust ratio to constrain the max attention logit more effectively. Moreover, we further construct element-wise trust ratios to provide more robust update scaling by focusing on local weight structures. Extensive experiments of large-batch training across various sizes of GPT-2 models demonstrate the superior performance of MERIT. Notably, during the training of GPT-2 Medium, MERIT enables a 6k batch size without any performance degradation compared to the standard batch size (480) with 48B training tokens. This work highlights the importance of considering the max attention logit and finer-granularity trust ratio in large-batch training. It successfully improves the training stability and paves the way for larger batch usage, enabling faster development and iteration of large language models. Code is available at https://github.com/NUS-HPC-AI-Lab/MERIT.
36	arxiv:2507.17501	2025	DNT: a Deeply Normalized Transformer that can be trained by Momentum SGD	Xianbiao Qi	arXiv.org	1	snowball-fwd-henry2020;snowball-fwd-vit22b;snowball-fwd-zhai2023	3	SCREEN	exclude [E5] title/abstract: not about attention-logit control or LM training stability	Transformers have become the de facto backbone of modern deep learning, yet their training typically demands an advanced optimizer with adaptive learning rate like AdamW, rather than a momentum SGDW (mSGDW). Previous works show that it is mainly due to a heavy-tailed distribution of the gradients. In this paper, we introduce a Deeply Normalized Transformer (DNT), which is meticulously engineered to overcome this limitation enabling seamless training with vanilla mSGDW while yielding comparable performance to the Transformers trained via AdamW. To be specific, in DNT, we strategically integrate normalization techniques at proper positions in the Transformers to effectively modulate the Jacobian matrices of each layer, balance the influence of weights, activations, and their interactions, and thus enable the distributions of gradients concentrated. We provide both theoretical justifications of the normalization technique used in our DNT and extensive empirical evaluation on two popular Transformer architectures to validate that: a) DNT outperforms its counterparts (\ie, ViT and GPT), and b) DNT can be effectively trained with vanilla mSGDW.
37	arxiv:2506.20752	2025	Characterization and Mitigation of Training Instabilities in Microscaling Formats	Huang-Yuan Su	arXiv.org	5	snowball-fwd-henry2020;snowball-fwd-vit22b;snowball-fwd-wortsman2024	3	SCREEN	exclude [E5] title/abstract: not about attention-logit control or LM training stability	Training large language models is an expensive, compute-bound process that must be repeated as models scale, algorithms improve, and new data is collected. To address this, next-generation hardware accelerators increasingly support lower-precision arithmetic formats, such as the Microscaling (MX) formats introduced in NVIDIA's Blackwell architecture. These formats use a shared scale within blocks of parameters to extend representable range and perform forward/backward GEMM operations in reduced precision for efficiency gains. In this work, we investigate the challenges and viability of block-scaled precision formats during model training. Across nearly one thousand language models trained from scratch -- spanning compute budgets from $2 \times 10^{17}$ to $4.8 \times 10^{19}$ FLOPs and sweeping over a broad range of weight-activation precision combinations -- we consistently observe that training in MX formats exhibits sharp, stochastic instabilities in the loss, particularly at larger compute scales. To explain this phenomenon, we conduct controlled experiments and ablations on a smaller proxy model that exhibits similar behavior as the language model, sweeping across architectural settings, hyperparameters, and precision formats. These experiments motivate a simple model in which multiplicative gradient bias introduced by the quantization of layer-norm affine parameters and a small fraction of activations can trigger runaway divergence. Through \emph{in situ} intervention experiments on our proxy model, we demonstrate that instabilities can be averted or delayed by modifying precision schemes mid-training. Guided by these findings, we evaluate stabilization strategies in the LLM setting and show that certain hybrid configurations recover performance competitive with full-precision training. We release our code at https://github.com/Hither1/systems-scaling.
38	arxiv:2503.04598	2025	HybridNorm: Towards Stable and Efficient Transformer Training via Hybrid Normalization	Zhijian Zhuo	Neural Information Processing Systems	25	snowball-fwd-henry2020;snowball-fwd-stability2024;snowball-fwd-vit22b	3	SCREEN	include [IA] HybridNorm: normalisation placement neighbour	Transformers have become the de facto architecture for a wide range of machine learning tasks, particularly in large language models (LLMs). Despite their remarkable performance, many challenges remain in training deep transformer networks, especially regarding the position of the layer normalization. While Pre-Norm structures facilitate more stable training owing to their stronger identity path, they often lead to suboptimal performance compared to Post-Norm. In this paper, we propose $\textbf{HybridNorm}$, a simple yet effective hybrid normalization strategy that integrates the advantages of both Pre-Norm and Post-Norm. Specifically, HybridNorm employs QKV normalization within the attention mechanism and Post-Norm in the feed-forward network (FFN) of each transformer block. We provide both theoretical insights and empirical evidence to demonstrate that HybridNorm improves the gradient flow and the model robustness. Extensive experiments on large-scale transformer models, including both dense and sparse variants, show that HybridNorm consistently outperforms both Pre-Norm and Post-Norm approaches across multiple benchmarks. These findings highlight the potential of HybridNorm as a more stable and effective technique for improving the training and performance of deep transformer models. Code is available at https://github.com/BryceZhuo/HybridNorm.
39	arxiv:2502.02732	2025	Peri-LN: Revisiting Normalization Layer in the Transformer Architecture	Jeonghoon Kim	International Conference on Machine Learning	41	snowball-fwd-vit22b;snowball-fwd-wortsman2024;snowball-fwd-zhai2023	3	SCREEN	include [IA] Peri-LN: normalisation placement neighbour	Selecting a layer normalization (LN) strategy that stabilizes training and speeds convergence in Transformers remains difficult, even for today's large language models (LLM). We present a comprehensive analytical foundation for understanding how different LN strategies influence training dynamics in large-scale Transformers. Until recently, Pre-LN and Post-LN have long dominated practices despite their limitations in large-scale training. However, several open-source models have recently begun silently adopting a third strategy without much explanation. This strategy places normalization layer peripherally around sublayers, a design we term Peri-LN. While Peri-LN has demonstrated promising performance, its precise mechanisms and benefits remain almost unexplored. Our in-depth analysis delineates the distinct behaviors of LN strategies, showing how each placement shapes activation variance and gradient propagation. To validate our theoretical insight, we conduct extensive experiments on Transformers up to $3.2$B parameters, showing that Peri-LN consistently achieves more balanced variance growth, steadier gradient flow, and convergence stability. Our results suggest that Peri-LN warrants broader consideration for large-scale Transformer architectures, providing renewed insights into the optimal placement of LN.
40	arxiv:2608.09160	2026	SwiftQK: Fast and Communication-Efficient Tensor Parallelism for Query-Key Normalization	Gyudong Kim	arXiv	0	A1-arxiv-qknorm;S1-openalex-qknorm;snowball-fwd-vit22b	3	SCREEN	exclude [E5] title/abstract: not about attention-logit control or LM training stability	Query-Key Normalization (QK-Norm) improves the training stability and quality of modern Large Language Models (LLMs). However, under Tensor Parallelism (TP), layerwise QK-Norm introduces additional cross-GPU communication because the normalization factor depends on the full hidden vector. We present SwiftQK, a multi-GPU RMSNorm kernel that exchanges only scalar normalization statistics and overlaps the remaining Peer-to-Peer reduction with independent element-wise computation in a deadlock-safe persistent kernel. Evaluations on recent LLMs show that SwiftQK reduces QK-Norm latency by 81.4--93.9% relative to the standard TP QK-Norm using full-vector All-Gather. In end-to-end serving, SwiftQK reduces TPOT on average by 29.5% over the All-Gather-based baseline and by 14.3% over an optimized scalar-aggregation implementation.
41	arxiv:2603.11611	2026	Fractional Rotation, Full Potential? Investigating Performance and Convergence of Partial RoPE	Mohammad Aflah Khan	arXiv	3	A1-arxiv-qknorm;S1-openalex-qknorm;snowball-fwd-henry2020	3	SCREEN	maybe [IC] NoPE loss spikes mitigated by QK-norm (side finding)	Rotary Positional Embedding (RoPE) is widely used in transformers to encode relative positional information, yet the effect of applying RoPE to only a subset of hidden dimensions remains underexplored. This design choice can substantially reduce memory usage, especially at long context lengths. We show that partial RoPE achieves up to 10$\times$ memory savings over standard RoPE cache while maintaining comparable final loss. We systematically study its impact on training dynamics and convergence across architectures, model sizes, sequence lengths, and datasets. Our results reveal that: (1) applying RoPE to only a small fraction of dimensions (around 10\%) matches the convergence of full RoPE; (2) these trends remain consistent across settings; and (3) NoPE (No Positional Encoding) can produce loss spikes, which are mitigated by minimal RoPE usage or QK-Norm application. Overall, our findings highlight partial RoPE as an effective trade-off between efficiency and stability.
42	arxiv:2602.08612	2026	OneLive: Dynamically Unified Generative Framework for Live-Streaming Recommendation	Shen Wang	arXiv	14	A1-arxiv-qknorm;S1-openalex-qknorm;snowball-fwd-henry2020	3	SCREEN	exclude [E5] title/abstract: not about attention-logit control or LM training stability	Live-streaming recommender system serves as critical infrastructure that bridges the patterns of real-time interactions between users and authors. Similar to traditional industrial recommender systems, live-streaming recommendation also relies on cascade architectures to support large-scale concurrency. Recent advances in generative recommendation unify the multi-stage recommendation process with Transformer-based architectures, offering improved scalability and higher computational efficiency. However, the inherent complexity of live-streaming prevents the direct transfer of these methods to live-streaming scenario, where continuously evolving content, limited lifecycles, strict real-time constraints, and heterogeneous multi-objectives introduce unique challenges that invalidate static tokenization and conventional model framework. To address these issues, we propose OneLive, a dynamically unified generative recommendation framework tailored for live-streaming scenario. OneLive integrates four key components: (i) A Dynamic Tokenizer that continuously encodes evolving real-time live content fused with behavior signal through residual quantization; (ii) A Time-Aware Gated Attention mechanism that explicitly models temporal dynamics for timely decision making; (iii) An efficient decoder-only generative architecture enhanced with Sequential MTP and QK Norm for stable training and accelerated inference; (iv) A Unified Multi-Objective Alignment Framework reinforces policy optimization for personalized preferences.
43	arxiv:2609.14715	2026	Depth and Scale in the Sub-150M Regime: JugnuLM-53M vs JugnuLM-110M	Dushyant Rajput	arXiv	0	A1-arxiv-qknorm;S1-openalex-qknorm;snowball-fwd-henry2020	3	SCREEN	maybe [IS] sub-150M depth study (QK-norm fixed, not varied)	We scale our conventional sub-150M pretraining recipe from 53.5M to 109.7M parameters, holding the method fixed (Qwen3-style decoder with grouped-query attention, RoPE, SwiGLU, RMSNorm, QK-Norm, and a z-loss; FineWeb-Edu data) and changing only the geometry to a deep-and-thin 23-layer x 576-hidden design. The larger model improves across the board -- BLiMP 78.1 -> 81.3, ARC-Easy 51.4 -> 52.5, WikiText-2 byte-perplexity 2.04 -> 1.95 -- and its 81.3% BLiMP essentially matches GPT-X2-125M (81.28) at about 12% fewer parameters. Notably the 110M model achieves this on fewer training tokens (about 8B vs 12B), so the gain is attributable to capacity and depth, not more data. Both models are deliberately conventional; this report is a clean scaling control and the baseline rung (R0) of an ablation study of what further improves models in this regime. An ablation ladder follows: value residuals (R1) and the Muon optimizer (R2) lift ARC-Easy by a cumulative +3.6 (52.5 -> 56.1) at a near-flat BLiMP and are kept; a diverse data blend (R3) and two logit-distillation settings (R4a/R4b) are not kept -- honest negatives. R3 pins ARC-Easy to FineWeb-Edu's educational filtering rather than raw diversity; distillation from a 1.7B teacher can reach the class-leading ARC-Easy (56.99, matching GPT-X2-125M) but only at a perplexity cost that dialing KD down then erases -- so R2 remains the best kept stack.
44	arxiv:2605.13989	2026	VectraYX-Nano: A 42M-Parameter Spanish Cybersecurity Language Model with Curriculum Learning and Native Tool Use	Juan S. Santillana	arXiv	4	A1-arxiv-qknorm;S1-openalex-qknorm;snowball-fwd-vit22b	3	SCREEN	exclude [E5] title/abstract: not about attention-logit control or LM training stability	We present VectraYX-Nano, a 41.95M-parameter decoder-only language model trained from scratch in Spanish for cybersecurity, with a Latin-American regional focus and native tool invocation via the Model Context Protocol (MCP). The model has four contributions. (i) Corpus: VectraYX-Sec-ES, a 170M-token Spanish corpus assembled by an eight-VM distributed pipeline at ~$25 USD of cloud compute and split into three curriculum phases (conversational 42M, cybersecurity 118M, offensive tooling 10M). (ii) Architecture: a 42M Transformer decoder with GQA, QK-Norm, RMSNorm, SwiGLU, RoPE and z-loss, paired with a domain-balanced 16,384-token byte-fallback BPE. (iii) Curriculum with replay across the three phases yields a monotonic loss descent (9.80 -> 3.17 -> 3.00 -> 2.16); after SFT (loss 1.74) the v2 bootstrap-ablation reference attains a conversational gate of 0.775 +/- 0.043 on B5 over N=4 seeds, and a controlled Phase-2 replay sweep over {0,5,10,25,50}% saturates B5 at >=25% replay. (iv) Two empirical findings, both N=4. A controlled bootstrap-corpus ablation across v2 (OpenSubs), v4 (mC4-ES), and v6 (60/25/15 OpenSubs/mC4/Wiki) exposes a loss-versus-register inversion: lower-perplexity bootstraps yield measurably worse conversational behavior (v2 > v4 > v6 on B5 at every paired seed). The B4 (tool-selection) floor of 0.000 is a corpus-density artifact, not a capacity gate: rebalancing the SFT mixture to tool-use ratio 1:21 yields VectraYX-Nano v7, the released headline configuration, reaching B4 = 0.230 +/- 0.052 at 42M while retaining B1 = 0.332 +/- 0.005 and B5 = 0.725 +/- 0.130; a LoRA replication on a 260M from-scratch mid-tier reaches 0.445 +/- 0.201. The released GGUF is 96 MB in F16, runs sub-second TTFT on commodity hardware under llama.cpp, and is, to our knowledge, the first published Spanish-native cybersecurity LLM with end-to-end MCP integration.
45	arxiv:2609.23314	2026	ValueDiff: Value-Geometric KV Cache Eviction for Sink-Suppressed LLMs	Junyoung Park	arXiv	0	A1-arxiv-qknorm;S1-openalex-qknorm;snowball-fwd-vit22b	3	SCREEN	exclude [E5] title/abstract: not about attention-logit control or LM training stability	Modern LLMs with QK-normalization, gated attention, learned attention sinks, or logit softcapping exhibit weaker persistent attention sinks, on which existing KV cache eviction methods primarily rely. We observe that across these models, weaker sinks co-occur with greater value-vector dispersion relative to key-vector dispersion. Motivated by this value-side dispersion, we present ValueDiff, a value-geometric eviction that ranks tokens by the L2 deviation of their value vectors from the cache mean. The same score arises as the minimal-disturbance eviction under a max-entropy assumption about future attention. We evaluate under fixed cache budgets, with eviction at every block boundary during prefill and at every decoding step during generation. On RULER at a tight 2k token budget, ValueDiff retains 88-99% of dense across seven sink-suppressed models (best on 6 out of 7). On LongBench at the 4k budget, ValueDiff averages 92% retention across sink-suppressed models versus 83% for the strongest prior baseline. On MATH-500, ValueDiff is the strongest non-dense method on every sink-suppressed model tested at the 25% cache budget, outperforming prior methods by up to ~20 points on gated-attention models. Across all three benchmarks, value geometry emerges as the more reliable query-invariant eviction signal for sink-suppressed models.
46	arxiv:2605.10504	2026	Learning Less Is More: Premature Upper-Layer Attention Specialization Hurts Language Model Pretraining	Jin-Chang Zhu	arXiv.org	2	snowball-fwd-henry2020;snowball-fwd-vit22b;snowball-fwd-zhai2023	3	SCREEN	exclude [E5] title/abstract: not about attention-logit control or LM training stability	A causal-decoder block is hierarchical: lower layers build the residual basis that upper layers attend over. We identify a failure mode in GPT pretraining: upper layers commit to sharp attention patterns before lower-layer features stabilize. We call this premature upper-layer attention specialization. Temporarily slowing only upper-layer Q/K projections during early training improves final perplexity and downstream accuracy without altering other parameters; it prevents upper attention from collapsing onto an immature residual basis. In LLaMA-style blocks, the same intervention is nearly unnecessary. Through ablations, we isolate multiplicative gated FFNs (not RMSNorm or bias removal) as the component that suppresses the upstream residual writes driving the failure. A pathwise analysis unifies both findings: the learning-rate intervention reduces a step-size factor, while gated FFNs reduce a residual-energy factor on the same growth pathway. Our results identify upper-layer Q/K timing as a concrete interaction point between decoder architecture and optimization.
47	arxiv:2605.06169	2026	Mean Mode Screaming: Mean-Variance Split Residuals for 1000-Layer Diffusion Transformers	Pengqian Lu	arXiv.org	0	snowball-fwd-henry2020;snowball-fwd-wortsman2024;snowball-fwd-zhai2023	3	SCREEN	exclude [E5] title/abstract: not about attention-logit control or LM training stability	Scaling Diffusion Transformers (DiTs) to hundreds of layers introduces a structural vulnerability: networks can enter a silent, mean-dominated collapse state that homogenizes token representations and suppresses centered variation. Through mechanistic auditing, we isolate the trigger event of this collapse as Mean Mode Screaming (MMS). MMS can occur even when training appears stable, with a mean-coherent backward shock on residual writers that opens deep residual branches and drives the network into a mean-dominated state. We show this behavior is driven by an exact decomposition of these gradients into mean-coherent and centered components, compounded by the structural suppression of attention-logit gradients through the null space of the Softmax Jacobian once values homogenize. To address this, we propose Mean-Variance Split (MV-Split) Residuals, which combine a separately gained centered residual update with a leaky trunk-mean replacement. On a 400-layer single-stream DiT, MV-Split prevents the divergent collapse that crashes the un-stabilized baseline; it tracks close to the baseline's pre-crash trajectory while remaining substantially better than token-isotropic gating methods such as LayerScale across the full schedule. Finally, we present a 1000-layer DiT as a scale-validation run at boundary scales, establishing that the architecture remains stably trainable at extreme depth.
48	arxiv:2601.22095	2026	GeoNorm: Unify Pre-Norm and Post-Norm with Geodesic Optimization	Chuan-Yang Zheng	arXiv.org	1	snowball-fwd-henry2020;snowball-fwd-stability2024;snowball-fwd-vit22b	3	SCREEN	exclude [E5] title/abstract: not about attention-logit control or LM training stability	The placement of normalization layers, specifically Pre-Norm and Post-Norm, remains an open question in Transformer architecture design. In this work, we rethink these approaches through the lens of manifold optimization, interpreting the outputs of the Feed-Forward Network (FFN) and attention layers as update directions in optimization. Building on this perspective, we introduce GeoNorm, a novel method that replaces standard normalization with geodesic updates on the manifold. Furthermore, analogous to learning rate schedules, we propose a layer-wise update decay for the FFN and attention components. Comprehensive experiments demonstrate that GeoNorm consistently outperforms existing normalization methods in Transformer models. Crucially, GeoNorm can be seamlessly integrated into standard Transformer architectures, achieving performance improvements with negligible additional computational cost.
49	arxiv:2606.08105	2026	A Unifying View of Attention Sinks: From Mechanisms to Architectural Interventions	Lukas Fesser	arXiv.org	2	snowball-fwd-stability2024;snowball-fwd-vit22b;snowball-fwd-zhai2023	3	SCREEN	exclude [E5] title/abstract: not about attention-logit control or LM training stability	When attention concentrates on a single token, a sink, what is the model actually computing? Attention sinks are ubiquitous in softmax transformers, yet this shared visual signature can hide fundamentally different algorithms. We show that visually similar sink patterns can reflect two distinct mechanisms: (i) adaptive nop, where a head suppresses its update by routing to a null token, and (ii) broadcast, where a sink aggregates and redistributes global information. Each mechanism leaves distinct traces (nop-sinks exhibit negligible value norms; broadcast sinks induce low-rank outputs), which we formalize on synthetic tasks and use to derive practical diagnostics. Applied to pretrained vision transformers, these diagnostics reveal that both mechanisms exist at scale: sinks transition from CLS in early layers to patches in deeper layers and concentrate in specialized heads. Causal interventions further connect these signatures to near-null suppression and shared residual contributions. We then use architectural interventions to show how these computations can be reorganized: gating eliminates detected nop-like sinks but increases broadcast-like sinks, registers relocate rather than remove sink computation, and our position-free global pathway provides an explicit route for shared communication that reduces the broadcast-like sinks induced by gating. On dense probes, combining gating with the global pathway gives the strongest results among the tested variants, despite retaining some broadcast-like sinks. Overall, we find that the same attention pattern can reflect two very different computations, and that effective intervention depends not only on identifying the computation, but also on providing architectural alternatives through which the model can reorganize it.
50	arxiv:2606.28116	2026	Mechanism-Aligned Observability for Training Instability of Large Language Models	Rui-Xuan Huang		1	snowball-fwd-vit22b;snowball-fwd-wortsman2024;snowball-fwd-zhai2023	3	SCREEN	maybe [IM] instability diagnostics; possible metrics	Frontier large language model training consumes massive accelerator fleets over long wall-clock times, making stability failures extremely costly. However, global symptoms such as loss divergences often emerge only after the underlying fault has already affected the training dynamics, further increasing the sunk compute. In which internal representation can a training fault become persistently and interpretably observable before it appears in global training metrics? We propose a mechanism-aligned principle for training observability: identify the internal state into which a fault is persistently written, derive the structure induced by its mechanism, and construct monitors in representations aligned with the computation of the affected module. For FlashAttention, we show that biased backward errors accumulate into a persistent low-dimensional structure in weight updates, producing spectral concentration. For MoE routers, we characterize routing dynamics through per-token routing behavior rather than parameter similarity, which may change without altering the routing function. Experiments show that the two monitors respond selectively to changes in their corresponding modules, before degradation becomes visible in global training metrics.
51	arxiv:2604.00199	2026	QUEST: A robust attention formulation using query-modulated spherical attention	Hariprasath Govindarajan	arXiv.org	1	snowball-fwd-vit22b;snowball-fwd-wortsman2024;snowball-fwd-zhai2023	3	SCREEN	exclude [E5] title/abstract: not about attention-logit control or LM training stability	The Transformer model architecture has become one of the most widely used in deep learning and the attention mechanism is at its core. The standard attention formulation uses a softmax operation applied to a scaled dot product between query and key vectors. We explore the role played by norms of the queries and keys, which can cause training instabilities when they arbitrarily increase. We demonstrate how this can happen even in simple Transformer models, in the presence of easy-to-learn spurious patterns in the data. We propose a new attention formulation, QUEry-modulated Spherical aTtention (QUEST), that constrains the keys to a hyperspherical latent space, while still allowing individual tokens to flexibly control the sharpness of the attention distribution. QUEST can be easily used as a drop-in replacement for standard attention. We focus on vision applications while also exploring other domains to highlight the method's generality. We show that (1) QUEST trains without instabilities and (2) produces models with improved performance (3) that are robust to data corruptions and adversarial attacks.
52	arxiv:2203.03466	2022	Tensor Programs V: Tuning Large Neural Networks via Zero-Shot Hyperparameter Transfer	Greg Yang	arXiv.org	340	snowball-back-wortsman2024;snowball-back-zhai2023	2	SCREEN	include [BG] muP: the confound held OFF	Hyperparameter (HP) tuning in deep learning is an expensive process, prohibitively so for neural networks (NNs) with billions of parameters. We show that, in the recently discovered Maximal Update Parametrization (muP), many optimal HPs remain stable even as model size changes. This leads to a new HP tuning paradigm we call muTransfer: parametrize the target model in muP, tune the HP indirectly on a smaller model, and zero-shot transfer them to the full-sized model, i.e., without directly tuning the latter at all. We verify muTransfer on Transformer and ResNet. For example, 1) by transferring pretraining HPs from a model of 13M parameters, we outperform published numbers of BERT-large (350M parameters), with a total tuning cost equivalent to pretraining BERT-large once; 2) by transferring from 40M parameters, we outperform published numbers of the 6.7B GPT-3 model, with tuning cost only 7% of total pretraining cost. A Pytorch implementation of our technique can be found at github.com/microsoft/mup and installable via `pip install mup`.
53	arxiv:2304.09871	2023	A Theory on Adam Instability in Large-Scale Machine Learning	Igor Molybog	arXiv.org	60	snowball-back-stability2024;snowball-back-wortsman2024	2	SCREEN	exclude [E5] title/abstract: not about attention-logit control or LM training stability	We present a theory for the previously unexplained divergent behavior noticed in the training of large language models. We argue that the phenomenon is an artifact of the dominant optimization algorithm used for training, called Adam. We observe that Adam can enter a state in which the parameter update vector has a relatively large norm and is essentially uncorrelated with the direction of descent on the training loss landscape, leading to divergence. This artifact is more likely to be observed in the training of a deep model with a large batch size, which is the typical setting of large-scale language model training. To argue the theory, we present observations from the training runs of the language models of different scales: 7 billion, 30 billion, 65 billion, and 546 billion parameters.
54	arxiv:2312.17482	2023	MosaicBERT: A Bidirectional Encoder Optimized for Fast Pretraining	J. Portes	Neural Information Processing Systems	47	snowball-fwd-henry2020;snowball-fwd-vit22b	2	SCREEN	exclude [E5] title/abstract: not about attention-logit control or LM training stability	Although BERT-style encoder models are heavily used in NLP research, many researchers do not pretrain their own BERTs from scratch due to the high cost of training. In the past half-decade since BERT first rose to prominence, many advances have been made with other transformer architectures and training configurations that have yet to be systematically incorporated into BERT. Here, we introduce MosaicBERT, a BERT-style encoder architecture and training recipe that is empirically optimized for fast pretraining. This efficient architecture incorporates FlashAttention, Attention with Linear Biases (ALiBi), Gated Linear Units (GLU), a module to dynamically remove padded tokens, and low precision LayerNorm into the classic transformer encoder block. The training recipe includes a 30% masking ratio for the Masked Language Modeling (MLM) objective, bfloat16 precision, and vocabulary size optimized for GPU throughput, in addition to best-practices from RoBERTa and other encoder models. When pretrained from scratch on the C4 dataset, this base model achieves a downstream average GLUE (dev) score of 79.6 in 1.13 hours on 8 A100 80 GB GPUs at a cost of roughly $20. We plot extensive accuracy vs. pretraining speed Pareto curves and show that MosaicBERT base and large are consistently Pareto optimal when compared to a competitive BERT base and large. This empirical speed up in pretraining enables researchers and engineers to pretrain custom BERT-style models at low cost instead of finetune on existing generic models. We open source our model weights and code.
55	arxiv:2312.16903	2023	Spike No More: Stabilizing the Pre-training of Large Language Models	Sho Takase	arXiv.org	56	snowball-fwd-wortsman2024;snowball-fwd-zhai2023	2	SCREEN	include [IA] pretraining loss-spike mitigation	Loss spikes often occur during pre-training of large language models. The spikes degrade the performance of large language models and sometimes ruin the pre-training. Since the pre-training needs a vast computational budget, we should avoid such spikes. Based on the assumption that the loss spike is caused by the sudden growth of the gradient norm, we explore factors to keep the gradient norm small through an analysis of the spectral norms of the Jacobian matrices for the sub-layers. Our findings suggest that stabilizing the pre-training process requires two conditions: small sub-layers and large shortcut. We conduct various experiments to empirically verify our theoretical analyses. Experimental results demonstrate that methods satisfying the conditions effectively prevent loss spikes during pre-training.
56	arxiv:2404.15702	2024	Nyonic Technical Report	Junfeng Tian	arXiv	0	A1-arxiv-qknorm;S1-openalex-qknorm	2	SCREEN	exclude [E5] title/abstract: not about attention-logit control or LM training stability	This report details the development and key achievements of our latest language model designed for custom large language models. The advancements introduced include a novel Online Data Scheduler that supports flexible training data adjustments and curriculum learning. The model's architecture is fortified with state-of-the-art techniques such as Rotary Positional Embeddings, QK-LayerNorm, and a specially crafted multilingual tokenizer to enhance stability and performance. Moreover, our robust training framework incorporates advanced monitoring and rapid recovery features to ensure optimal efficiency. Our Wonton 7B model has demonstrated competitive performance on a range of multilingual and English benchmarks. Future developments will prioritize narrowing the performance gap with more extensively trained models, thereby enhancing the model's real-world efficacy and adaptability.GitHub: \url{https://github.com/nyonicai/nyonic-public}
57	arxiv:2407.01800	2024	Normalization and effective learning rates in reinforcement learning	Clare Lyle	Neural Information Processing Systems	70	snowball-fwd-henry2020;snowball-fwd-wortsman2024	2	SCREEN	exclude [E5] title/abstract: not about attention-logit control or LM training stability	Normalization layers have recently experienced a renaissance in the deep reinforcement learning and continual learning literature, with several works highlighting diverse benefits such as improving loss landscape conditioning and combatting overestimation bias. However, normalization brings with it a subtle but important side effect: an equivalence between growth in the norm of the network parameters and decay in the effective learning rate. This becomes problematic in continual learning settings, where the resulting effective learning rate schedule may decay to near zero too quickly relative to the timescale of the learning problem. We propose to make the learning rate schedule explicit with a simple re-parameterization which we call Normalize-and-Project (NaP), which couples the insertion of normalization layers with weight projection, ensuring that the effective learning rate remains constant throughout training. This technique reveals itself as a powerful analytical tool to better understand learning rate schedules in deep reinforcement learning, and as a means of improving robustness to nonstationarity in synthetic plasticity loss benchmarks along with both the single-task and sequential variants of the Arcade Learning Environment. We also show that our approach can be easily applied to popular architectures such as ResNets and transformers while recovering and in some cases even slightly improving the performance of the base model in common stationary benchmarks.
58	arxiv:2406.06248	2024	Compute Better Spent: Replacing Dense Layers with Structured Matrices	Shikai Qiu	International Conference on Machine Learning	37	snowball-fwd-henry2020;snowball-fwd-wortsman2024	2	SCREEN	exclude [E5] title/abstract: not about attention-logit control or LM training stability	Dense linear layers are the dominant computational bottleneck in foundation models. Identifying more efficient alternatives to dense matrices has enormous potential for building more compute-efficient models, as exemplified by the success of convolutional networks in the image domain. In this work, we systematically explore structured matrices as replacements for dense matrices. We show that different structures often require drastically different initialization scales and learning rates, which are crucial to performance, especially as models scale. Using insights from the Maximal Update Parameterization, we determine the optimal scaling for initialization and learning rates of these unconventional layers. Finally, we measure the scaling laws of different structures to compare how quickly their performance improves with compute. We propose a novel matrix family containing Monarch matrices, the Block Tensor-Train (BTT), which we show performs better than dense matrices for the same compute on multiple tasks. On CIFAR-10/100 with augmentation, BTT achieves exponentially lower training loss than dense when training MLPs and ViTs. BTT matches dense ViT-S/32 performance on ImageNet-1k with 3.8 times less compute and is more efficient than dense for training small GPT-2 language models.
59	arxiv:2405.05945	2024	Lumina-T2X: Transforming Text into Any Modality, Resolution, and Duration via Flow-based Large Diffusion Transformers	Peng Gao	arXiv.org	147	snowball-fwd-henry2020;snowball-fwd-vit22b	2	SCREEN	exclude [E5] title/abstract: not about attention-logit control or LM training stability	Sora unveils the potential of scaling Diffusion Transformer for generating photorealistic images and videos at arbitrary resolutions, aspect ratios, and durations, yet it still lacks sufficient implementation details. In this technical report, we introduce the Lumina-T2X family - a series of Flow-based Large Diffusion Transformers (Flag-DiT) equipped with zero-initialized attention, as a unified framework designed to transform noise into images, videos, multi-view 3D objects, and audio clips conditioned on text instructions. By tokenizing the latent spatial-temporal space and incorporating learnable placeholders such as [nextline] and [nextframe] tokens, Lumina-T2X seamlessly unifies the representations of different modalities across various spatial-temporal resolutions. This unified approach enables training within a single framework for different modalities and allows for flexible generation of multimodal data at any resolution, aspect ratio, and length during inference. Advanced techniques like RoPE, RMSNorm, and flow matching enhance the stability, flexibility, and scalability of Flag-DiT, enabling models of Lumina-T2X to scale up to 7 billion parameters and extend the context window to 128K tokens. This is particularly beneficial for creating ultra-high-definition images with our Lumina-T2I model and long 720p videos with our Lumina-T2V model. Remarkably, Lumina-T2I, powered by a 5-billion-parameter Flag-DiT, requires only 35% of the training computational costs of a 600-million-parameter naive DiT. Our further comprehensive analysis underscores Lumina-T2X's preliminary capability in resolution extrapolation, high-resolution editing, generating consistent 3D views, and synthesizing videos with seamless transitions. We expect that the open-sourcing of Lumina-T2X will further foster creativity, transparency, and diversity in the generative AI community.
60	arxiv:2410.24050	2024	A Mechanistic Study of Transformers Training Dynamics	Ambroise Odonnat		2	snowball-fwd-vit22b;snowball-fwd-wortsman2024	2	SCREEN	exclude [E5] title/abstract: not about attention-logit control or LM training stability	Large-scale pretraining of transformers has been central to the success of foundation models. However, the scale of those models limits our understanding of the mechanisms at play during optimization. In this work, we study the training dynamics of transformers in a controlled and interpretable setting. On the sparse modular addition task, we demonstrate that specialized attention circuits, called clustering heads, can be implemented during gradient descent to solve the problem. Our experiments show that such pathways naturally emerge during training. By monitoring the evolution of tokens via a visual sandbox, we uncover a two-stage learning and the occurrences of loss spikes due to the high curvature of normalization layers. Our findings provide several insights into patterns observed in more practical settings, such as the pretraining of large language models.
61	arxiv:2407.07972	2024	Deconstructing What Makes a Good Optimizer for Language Models	Rosie Zhao	International Conference on Learning Representations	48	snowball-fwd-vit22b;snowball-fwd-wortsman2024	2	SCREEN	maybe [BG] LR stability across optimisers, small LMs	Training language models becomes increasingly expensive with scale, prompting numerous attempts to improve optimization efficiency. Despite these efforts, the Adam optimizer remains the most widely used, due to a prevailing view that it is the most effective approach. We aim to compare several optimization algorithms, including SGD, Adafactor, Adam, Lion, and Sophia in the context of autoregressive language modeling across a range of model sizes, hyperparameters, and architecture variants. Our findings indicate that, except for SGD, these algorithms all perform comparably both in their optimal performance and also in terms of how they fare across a wide range of hyperparameter choices. Our results suggest to practitioners that the choice of optimizer can be guided by practical considerations like memory constraints and ease of implementation, as no single algorithm emerged as a clear winner in terms of performance or stability to hyperparameter misspecification. Given our findings, we further dissect these approaches, examining two simplified versions of Adam: a) signed momentum (Signum) which we see recovers both the performance and hyperparameter stability of Adam and b) Adalayer, a layerwise variant of Adam which we introduce to study the impact on Adam's preconditioning for different layers of the network. Examining Adalayer leads us to the conclusion that, perhaps surprisingly, adaptivity on both the last layer and LayerNorm parameters in particular are necessary for retaining performance and stability to learning rate.
62	arxiv:2406.15303	2024	AEM: Attention Entropy Maximization for Multiple Instance Learning Based Whole Slide Image Classification	Yunlong Zhang	International Conference on Medical Image Computing and Computer-Assisted Intervention	16	snowball-fwd-vit22b;snowball-fwd-zhai2023	2	SCREEN	exclude [E5] title/abstract: not about attention-logit control or LM training stability	Multiple Instance Learning (MIL) effectively analyzes whole slide images but faces overfitting due to attention over-concentration. While existing solutions rely on complex architectural modifications or additional processing steps, we introduce Attention Entropy Maximization (AEM), a simple yet effective regularization technique. Our investigation reveals the positive correlation between attention entropy and model performance. Building on this insight, we integrate AEM regularization into the MIL framework to penalize excessive attention concentration. To address sensitivity to the AEM weight parameter, we implement Cosine Weight Annealing, reducing parameter dependency. Extensive evaluations demonstrate AEM's superior performance across diverse feature extractors, MIL frameworks, attention mechanisms, and augmentation techniques. Here is our anonymous code: https://github.com/dazhangyu123/AEM.
63	arxiv:2405.13698	2024	How to set AdamW's weight decay as you scale model and dataset size	Xi Wang	International Conference on Machine Learning	52	snowball-fwd-vit22b;snowball-fwd-wortsman2024	2	SCREEN	maybe [BG] AdamW hyperparameters vs scale	The scaling of the optimal AdamW weight decay hyperparameter with model and dataset size is critical as we seek to build larger models, but is poorly understood. We show that weights learned by AdamW can be understood as an exponential moving average (EMA) of recent updates. This gives critical insights for how to set the weight decay in AdamW, and how the weight decay should scale with model and dataset size. In particular, the key hyperparameter for an exponential moving average is the EMA timescale. Intuitively, the EMA timescale can be understood as the number of recent iterations the EMA averages over. We find that the optimal timescale, measured in epochs, is roughly constant as we change model and dataset size. Moreover, given a learning rate, there is a one-to-one mapping from the EMA timescale to the weight decay hyperparameter. Thus, if the optimal EMA timescale is constant, that implies that as the dataset size increases, the optimal weight decay should fall and as the model size increases, the optimal weight decay should increase (if we follow the muP recommendation for scaling the learning rate). We validate these scaling rules on ResNet-18 and Vision Transformers trained on CIFAR-10 and ImageNet, and on NanoGPT pre-training on OpenWebText. Finally, we found that as training progresses, muP's learning rate scaling breaks down for AdamW unless weight decay is scaled appropriately.
64	arxiv:2405.18710	2024	To FP8 and Back Again: Quantifying the Effects of Reducing Precision on LLM Training Stability	Joonhyung Lee	arXiv.org	18	snowball-fwd-vit22b;snowball-fwd-wortsman2024	2	SCREEN	exclude [E5] title/abstract: not about attention-logit control or LM training stability	The massive computational costs associated with large language model (LLM) pretraining have spurred great interest in reduced-precision floating-point representations to accelerate the process. As a result, the BrainFloat16 (BF16) precision has become the de facto standard for LLM training, with hardware support included in recent generations of accelerators. This trend has gone even further in the latest processors, where FP8 has recently been introduced. However, prior experience with FP16, which was found to be less stable than BF16, raises concerns as to whether FP8, with even fewer bits than FP16, can be a cost-effective option for LLM training. We argue that reduced-precision training schemes must have similar training stability and hyperparameter sensitivities to their higher-precision counterparts in order to be cost-effective. However, we find that currently available methods for FP8 training are not robust enough to allow their use as economical replacements. This prompts us to investigate the stability of reduced-precision LLM training in terms of robustness across random seeds, learning rates, and datasets. To this end, we propose new evaluation techniques and a new metric for quantifying loss landscape sharpness in autoregressive language models. By simulating incremental bit reductions in floating-point representations, we analyze the relationship between representational power and training stability with the intent of aiding future research into the field.
65	arxiv:2411.00999	2024	Normalization Layer Per-Example Gradients are Sufficient to Predict Gradient Noise Scale in Transformers	Gavia Gray	Neural Information Processing Systems	7	snowball-fwd-wortsman2024;snowball-fwd-zhai2023	2	SCREEN	exclude [E5] title/abstract: not about attention-logit control or LM training stability	Per-example gradient norms are a vital ingredient for estimating gradient noise scale (GNS) with minimal variance. Observing the tensor contractions required to compute them, we propose a method with minimal FLOPs in 3D or greater tensor regimes by simultaneously computing the norms while computing the parameter gradients. Using this method we are able to observe the GNS of different layers at higher accuracy than previously possible. We find that the total GNS of contemporary transformer models is predicted well by the GNS of only the normalization layers. As a result, focusing only on the normalization layer, we develop a custom kernel to compute the per-example gradient norms while performing the LayerNorm backward pass with zero throughput overhead. Tracking GNS on only those layers, we are able to guide a practical batch size schedule that reduces training time by 18% on a Chinchilla-optimal language model.
66	arxiv:2510.09016	2025	DiTSinger: Scaling Singing Voice Synthesis with Diffusion Transformer and Implicit Alignment	Zongcai Du	arXiv	0	A1-arxiv-qknorm;S1-openalex-qknorm	2	SCREEN	exclude [E5] title/abstract: not about attention-logit control or LM training stability	Recent progress in diffusion-based Singing Voice Synthesis (SVS) demonstrates strong expressiveness but remains limited by data scarcity and model scalability. We introduce a two-stage pipeline: a compact seed set of human-sung recordings is constructed by pairing fixed melodies with diverse LLM-generated lyrics, and melody-specific models are trained to synthesize over 500 hours of high-quality Chinese singing data. Building on this corpus, we propose DiTSinger, a Diffusion Transformer with RoPE and qk-norm, systematically scaled in depth, width, and resolution for enhanced fidelity. Furthermore, we design an implicit alignment mechanism that obviates phoneme-level duration labels by constraining phoneme-to-acoustic attention within character-level spans, thereby improving robustness under noisy or uncertain alignments. Extensive experiments validate that our approach enables scalable, alignment-free, and high-fidelity SVS.
67	arxiv:2505.24333	2025	Two failure modes of deep transformers and how to avoid them: a unified theory of signal propagation at initialisation	Alessio Giorlandino	arXiv	17	A2-arxiv-logit-instability;snowball-fwd-zhai2023	2	SCREEN	include [IM] rank vs entropy collapse in deep transformers (depth)	Finding the right initialisation for neural networks is crucial to ensure smooth training and good performance. In transformers, the wrong initialisation can lead to one of two failure modes of self-attention layers: rank collapse, where all tokens collapse into similar representations, and entropy collapse, where highly concentrated attention scores lead to training instability. While previous work has studied different scaling regimes for transformers, an asymptotically exact, down-to-the constant prescription for how to initialise transformers has so far been lacking. Here, we provide an analytical theory of signal propagation through deep transformers with self-attention, layer normalisation, skip connections and MLP. Our theory yields a simple algorithm to compute trainability diagrams that identify the correct choice of initialisation hyper-parameters for a given architecture. We overcome the key challenge, an exact treatment of the self-attention layer, by establishing a formal parallel with the Random Energy Model from statistical physics. We also analyse gradients in the backward path and determine the regime where gradients vanish at initialisation. We demonstrate the versatility of our framework through three case studies. Our theoretical framework gives a unified perspective on the two failure modes of self-attention and gives quantitative predictions on the scale of both weights and residual connections that guarantee smooth training.
68	doi:10.1109/access.2025.3601206	2025	A Systematic Literature Review of Hallucinations in Large Language Models	Christian Woesle	IEEE Access	22	L1-openalex-llm-for-lit-search;L2-openalex-llm-slr-software-eng	2	SCREEN	exclude [E5] title/abstract: not about attention-logit control or LM training stability	This review systematically maps research on hallucinations in large language models using a descriptive scheme that links model outputs to four system architectures: unaugmented generation, post-hoc reactive validation, proactive detection-and-mitigation, and fully integrated detection-and-mitigation designs. Our methodology for this systematic review follows the PRISMA guidelines to ensure transparency and reproducibility. We searched IEEE Xplore, ACM Digital Library, and ScienceDirect for studies published between 2015 and January 2025 and extracted 125 peer-reviewed papers across nine application domains. Quantitative analysis shows that question answering and multimodal tasks account for 48% of all papers, whereas software engineering, educational technology, and autonomous systems are underexplored. Although 87.5% of the studies rely on additional reactive or proactive defenses, only 8.8% implement integrated architecture-level safeguards, revealing a critical gap in unified and dynamic architectures. The resulting classification matrix and domain map provide a diagnostic tool for locating blind spots and comparing architectural maturity. Three actionable priorities emerge: develop integrated reasoning-and-verification loops that pre-empt hallucinations; transfer proven causal-intervention and multi-agent validation pipelines to high-stakes, under-represented domains and benchmark them under real conditions; and build modular, cross-domain evaluation frameworks that isolate the contribution of individual mitigation components and support ablation studies. By consolidating fragmented evidence and quantifying architecture-domain imbalances, this review establishes a traceable foundation for engineering reliable, explainable, and domain-adaptable countermeasures to hallucinations in generative language technology.
69	doi:10.5335/rbca.v17i3.16310	2025	A Systematic Review about Large Language Models (LLMs) applied to Code Generation	Fabio Alecsandro Bacin	Revista Brasileira de Computação Aplicada	1	L1-openalex-llm-for-lit-search;L2-openalex-llm-slr-software-eng	2	SCREEN	exclude [E5] title/abstract: not about attention-logit control or LM training stability	Large Language Models (LLMs) for code generation represent a significant advancement in software development by boosting productivity, simplifying repetitive tasks, enabling automated testing, and promoting best practices. This paper presents a systematic literature review (SLR) of studies focused on LLMs applied to code generation. The review enhances understanding of the capabilities and limitations of these models, outlining both their benefits and challenges. The review protocol included a search on the Google Scholar database using relevant keywords related to LLMs and code generation. A total of 112 articles were initially retrieved, from which 15 were selected based on relevance and quality criteria. Of these, 8 were analyzed in depth to evaluate various approaches and outcomes, while the remaining 7 provided the theoretical foundation for the study. This work contributes to the growing body of knowledge in the field and supports future research and applications of LLMs in software engineering.
70	arxiv:2512.22382	2025	Completed Hyperparameter Transfer across Modules, Width, Depth, Batch and Duration	Bruno Mlodozeniec	arXiv.org	26	snowball-fwd-henry2020;snowball-fwd-vit22b	2	SCREEN	exclude [E5] title/abstract: not about attention-logit control or LM training stability	Hyperparameter tuning can dramatically impact training stability and final performance of large-scale models. Recent works on neural network parameterisations, such as $\mu$P, have enabled transfer of optimal global hyperparameters across model sizes. These works propose an empirical practice of search for optimal global base hyperparameters at a small model size, and transfer to a large size. We extend these works in two key ways. To handle scaling along most important scaling axes, we propose the Complete$^{(d)}$ Parameterisation that unifies scaling in width and depth -- using an adaptation of CompleteP -- as well as in batch-size and training duration. Secondly, with our parameterisation, we investigate per-module hyperparameter optimisation and transfer. We characterise the empirical challenges of navigating the high-dimensional hyperparameter landscape, and propose practical guidelines for tackling this optimisation problem. We demonstrate that, with the right parameterisation, hyperparameter transfer holds even in the per-module hyperparameter regime. Our study covers an extensive range of optimisation hyperparameters of modern models: learning rates, AdamW parameters, weight decay, initialisation scales, and residual block multipliers. Our experiments demonstrate significant training speed improvements in Large Language Models with the transferred per-module hyperparameters.
71	arxiv:2505.20524	2025	Towards Fully FP8 GEMM LLM Training at Scale	Alejandro Hernández-Cano	Neural Information Processing Systems	13	snowball-fwd-henry2020;snowball-fwd-zhai2023	2	SCREEN	exclude [E5] title/abstract: not about attention-logit control or LM training stability	Despite the significant potential of FP8 data formats for large language model (LLM) pre-training, their adoption has been limited due to challenges in maintaining stability at scale. Existing approaches often rely on suboptimal fine-grained FP8 kernels or fall back to higher-precision matrix multiplications (GEMMs) in sensitive components, such as attention projections, compromising potential throughput gains. We introduce a new class of LLM architectures that, for the first time, support FP8 computation for all GEMMs within transformer blocks during both forward and backward passes. This enables unprecedented throughput gains, particularly at scale, while matching the downstream performance of standard BF16 training. Our architecture design reduces large outlier activations, promoting stable long-term FP8 training. In addition, we identify key metrics to monitor low-precision training and predict potential future divergences.
72	arxiv:2501.13428	2025	Softplus Attention with Re-weighting Boosts Length Extrapolation in Large Language Models	Bo Gao	arXiv.org	1	snowball-fwd-henry2020;snowball-fwd-vit22b	2	SCREEN	exclude [E5] title/abstract: not about attention-logit control or LM training stability	Large language models have achieved remarkable success in recent years, primarily due to self-attention. However, traditional Softmax attention suffers from numerical instability and reduced performance as the number of inference tokens increases. This work addresses these issues by proposing a new design principle for attention, viewing it as a two-stage process. The first stage (normalisation) refines standard attention by replacing Softmax with the more numerically stable Softplus followed by $l_{1}$-normalisation. Furthermore, we introduce a dynamic scale factor based on invariance entropy. We show that this novel attention mechanism outperforms conventional Softmax attention, and state-of-the-art Softmax-free alternatives. Our second proposal is to introduce a second processing stage (sharpening) which consists of a re-weighting mechanism that amplifies significant attentional weights while diminishing weaker ones. This enables the model to concentrate more effectively on relevant tokens, mitigating the attention sink phenomenon, and fundamentally improving length extrapolation. This novel, two-stage, replacement for self-attention is shown to ensure numerical stability and dramatically improve length extrapolation, maintaining a nearly constant validation loss at 16$\times$ the training length while achieving superior results on challenging long-context retrieval tasks and downstream benchmarks. Furthermore, symbolic regression experiments demonstrate that our method enables models to recover Newton's gravitational law from orbital trajectory sequences, providing evidence that appropriate attention mechanisms are crucial for foundation models to develop genuine physical world models. Our code is available at https://github.com/iminfine/freeattn.
73	arxiv:2510.04202	2025	Spectral Alignment as Predictor of Loss Explosion in Neural Network Training	Hai-Quan Qiu	arXiv.org	2	snowball-fwd-stability2024;snowball-fwd-wortsman2024	2	SCREEN	exclude [E5] title/abstract: not about attention-logit control or LM training stability	Loss explosions in training deep neural networks can nullify multi-million dollar training runs. Conventional monitoring metrics like weight and gradient norms are often lagging and ambiguous predictors, as their values vary dramatically across different models and even between layers of the same model, making it difficult to establish a unified standard for detecting impending failure. We introduce Spectral Alignment (SA), a novel, theoretically-grounded metric that monitors the distributional alignment between layer inputs and the principal singular vectors of weight matrices. We show that a collapse in the sign diversity of this alignment is a powerful early predictor of representational collapse and training divergence. Empirical results on language models demonstrate that monitoring the SA distribution provides a significantly earlier and clearer warning of loss explosions than traditional scalar metrics. SA's low computational overhead makes it a practical tool for safeguarding model training.
74	arxiv:2507.20534	2025	Kimi K2: Open Agentic Intelligence	Kimi Team Yifan Bai		439	snowball-fwd-vit22b;snowball-fwd-wortsman2024	2	SCREEN	include [IA] MuonClip/QK-clip: alternative fix for attention-logit explosion	We introduce Kimi K2, a Mixture-of-Experts (MoE) large language model with 32 billion activated parameters and 1 trillion total parameters. We propose the MuonClip optimizer, which improves upon Muon with a novel QK-clip technique to address training instability while enjoying the advanced token efficiency of Muon. Based on MuonClip, K2 was pre-trained on 15.5 trillion tokens with zero loss spike. During post-training, K2 undergoes a multi-stage post-training process, highlighted by a large-scale agentic data synthesis pipeline and a joint reinforcement learning (RL) stage, where the model improves its capabilities through interactions with real and synthetic environments. Kimi K2 achieves state-of-the-art performance among open-source non-thinking models, with strengths in agentic capabilities. Notably, K2 obtains 66.1 on Tau2-Bench, 76.5 on ACEBench (En), 65.8 on SWE-Bench Verified, and 47.3 on SWE-Bench Multilingual -- surpassing most open and closed-sourced baselines in non-thinking settings. It also exhibits strong capabilities in coding, mathematics, and reasoning tasks, with a score of 53.7 on LiveCodeBench v6, 49.5 on AIME 2025, 75.1 on GPQA-Diamond, and 27.1 on OJBench, all without extended thinking. These results position Kimi K2 as one of the most capable open-source large language models to date, particularly in software engineering and agentic tasks. We release our base and post-trained model checkpoints to facilitate future research and applications of agentic intelligence.
75	arxiv:2502.11034	2025	AdaGC: Enhancing LLM Pretraining Stability via Adaptive Gradient Clipping	Guo-Xia Wang	arXiv.org	8	snowball-fwd-vit22b;snowball-fwd-wortsman2024	2	SCREEN	exclude [E5] title/abstract: not about attention-logit control or LM training stability	Loss spikes remain a persistent obstacle in large-scale language model pretraining. While previous research has attempted to identify the root cause of loss spikes by investigating individual factors, we observe that, in practice, such spikes are typically triggered by the confluence of heterogeneous factors. Empirically, loss spikes may arise from a combination of data outliers, hardware or transient computational faults, numerical precision issues, and hyperparameter settings. Regardless of the underlying cause, these spikes manifest as unstable optimizer updates, as abnormal gradients contaminate both first- and second-moment states. In this paper, we propose a principled gradient-centric remedy: AdaGC, an adaptive per-tensor gradient clipping scheme that mitigates such contamination by bounding gradient norms relative to a tensor-wise exponential moving average of their historical clipped values. AdaGC is optimizer-agnostic, introduces negligible memory overhead, and reduces communication costs compared to GlobalGC, particularly in hybrid-parallel distributed training. Experiments on Llama-2 7B, Mixtral 8x1B, and ERNIE 10B-A1.4B demonstrate that AdaGC robustly eliminates training instabilities, consistently reducing spike scores to zero for all models and improving downstream accuracy over GlobalGC by 1.32%, 1.27%, and 2.48%, respectively. Furthermore, AdaGC seamlessly integrates with optimizers such as Muon and Lion, consistently yielding higher average accuracy and zero spike scores. The code is available at https://github.com/PaddlePaddle/PaddleFleet (see Research/AdaGC).
76	arxiv:2501.18537	2025	Loss Functions and Operators Generated by f-Divergences	Vincent Roulet	International Conference on Machine Learning	15	snowball-fwd-vit22b;snowball-fwd-wortsman2024	2	SCREEN	exclude [E5] title/abstract: not about attention-logit control or LM training stability	The logistic loss (a.k.a. cross-entropy loss) is one of the most popular loss functions used for multiclass classification. It is also the loss function of choice for next-token prediction in language modeling. It is associated with the Kullback--Leibler (KL) divergence and the softargmax operator. In this work, we propose to construct new convex loss functions based on $f$-divergences. Our loss functions generalize the logistic loss in two directions: i) by replacing the KL divergence with $f$-divergences and ii) by allowing non-uniform reference measures. We instantiate our framework for numerous $f$-divergences, recovering existing losses and creating new ones. By analogy with the logistic loss, the loss function generated by an $f$-divergence is associated with an operator, that we dub $f$-softargmax. We derive a novel parallelizable bisection algorithm for computing the $f$-softargmax associated with any $f$-divergence. On the empirical side, one of the goals of this paper is to determine the effectiveness of loss functions beyond the classical cross-entropy in a language model setting, including on pre-training, post-training (SFT) and distillation. We show that the loss function generated by the $\alpha$-divergence (which is equivalent to Tsallis $\alpha$-negentropy in the case of unit reference measures) with $\alpha=1.5$ performs well across several tasks.
77	arxiv:2610.17969	2026	Neural Dimensional Analysis: Foundations of Dimensionless and Scale-Consistent Computation	Bum Jun Kim	arXiv	0	A1-arxiv-qknorm;S1-openalex-qknorm	2	SCREEN	include [IA*] compares vs QK-Norm across 16x LR range at 124M	Modern Transformers rely on many scale-control mechanisms, including normalization, residual scaling, attention temperatures, loss weighting, and adaptive updates. To explain how these mechanisms maintain scale consistency, we introduce Neural Dimensional Analysis (NDA), a unified framework based on dimensional consistency. We define neural units that specify how tensors rescale and which quantities can be added, just as physical units do in physics. Our rules match residual units, nondimensionalize attention and gate inputs, combine losses across units, and convert gradient units into parameter units. We implement NDA through dimensionless Unit Violation Indices (UVI) for diagnosing scale mismatches within Transformers, a unit-consistent multi-loss objective, and the Gauge-Canonical Transformer (GCT) with normalized coordinates and orthogonalized residual additions. In language-model experiments spanning 51M to 1B parameters, GCT remains stable during training across a broad sixteenfold learning-rate range, outperforming Pre-LN, LLaMA-style, and QK-Norm in 124M validation loss at tuned learning rates. GCT's normalized residual stream also shows improved robustness to quantization. At 1B parameters, calibrated per-tensor int8 quantization increases GCT's validation loss by only 0.009 nats, compared with up to 5.51 nats for the baselines. Vision and physics-informed neural network experiments further confirm NDA's benefits beyond language modeling. These results establish dimensional consistency as a neural network design principle.
78	arxiv:2605.18898	2026	A Two-Parameter Weibull Framework for Diagnosing Transformer Weight Distributions	Tiexin Ding	arXiv	0	A1-arxiv-qknorm;S1-openalex-qknorm	2	SCREEN	exclude [E5] title/abstract: not about attention-logit control or LM training stability	We apply the Weibull distribution -- a two-parameter family from extreme-value theory -- as a diagnostic framework for element-wise weight magnitude distributions in transformers. At initialization, i.i.d. Gaussian weights give \|w\| ~ HalfNormal, yielding k ~ 1.20 via middle-80% probability-plot fit (the protocol used throughout this work). This anchor makes k a principled, architecture-independent measuring stick for training dynamics; fitting each weight matrix independently at every layer at every checkpoint enables per-component, per-layer, and per-step diagnostics that aggregate statistics cannot resolve. Applying this framework to 12 model entries spanning 7 architectural families (Pythia, OLMo-1/2, LLaMA-3, Mistral, Qwen2.5/3) reveals three findings. First, FFN modules and the attention output projection W_o -- the Transmission Class -- fall in a narrow k band: median terminal k in [1.186, 1.204] across 12 entries (cross-family CV = 0.51%), shared across SwiGLU/GeLU activations, Pre-LN/QK-Norm placements, and 70M-14B sizes. Second, the attention input projections W_q, W_k -- the Selection Class -- depart from the Weibull family, with severity shaped by storage: separately-stored Q/K (OLMo-1, OLMo-2) yields k in [0.76, 0.99] (deep); GQA models yield k in [1.10, 1.16] (mild); Pythia's merged W_qkv occupies a transitional zone tracking training budget T/tau monotonically. Third, lambda grows substantially during training and scales with sqrt(eta/lambda_wd) within the Pythia family (Pearson r = 0.94, three Transmission kinds), directionally consistent with Fan et al. (2025). The two parameters carry independent information: k labels the functional class, lambda labels training progress. We release npm-weibull-py v0.4 (Python library) and DATABASE_v9_1 at https://github.com/tiexinding/NPM-Weibull-public .
79	arxiv:2609.37852	2026	Delta-Matching: Closing the Final Gap of Native 8-bit Training for LLMs	Haozhan Tang	arXiv	0	A1-arxiv-qknorm;S1-openalex-qknorm	2	SCREEN	exclude [E5] title/abstract: not about attention-logit control or LM training stability	Reliable FP8 attention remains a barrier to fully native 8-bit large language model training. We derive how forward-backward inconsistencies produce stale delta and empirically show how it distorts training dynamics. Our stale-delta hybrid runs show a modest loss gap at 569M parameters but substantial loss increases and downstream degradation at 1.67B and 5.29B. QK normalization, NoPE (no positional encoding), and lower-learning-rate context extension mitigate or delay degradation without eliminating it. This pattern suggests accumulated optimization error that smaller models and short runs can conceal. We propose Delta-Matching, proving that it restores the softmax gradient's zero-row-sum invariant under the stated numerical assumptions. It enables native block-scaled FP8 in every forward and backward attention-core matmul without architectural changes, smaller global batches, or auxiliary forward outputs. Across tested architectures, scales, and training stages, Delta-Matching matches BF16/FP32 mixed-precision training loss and overall downstream performance. We will release our implementation, trained models, and data recipes.
80	arxiv:2606.20027	2026	QG-MIL: A Gated Transformer Aggregator for Domain-Agnostic Multiple Instance Learning in Medical Imaging	Luca Zedda	arXiv	0	A1-arxiv-qknorm;S1-openalex-qknorm	2	SCREEN	exclude [E5] title/abstract: not about attention-logit control or LM training stability	Attention-based Multiple Instance Learning aggregators in medical imaging are prone to attention concentration, producing overconfident and unstable predictions. We introduce QG-MIL, a gated transformer aggregator that addresses this through four synergistic architectural components: RMSNorm-based pre-normalization, per-head QK normalization, fine-grained attention output gating, and SwiGLU-style feed-forward modules. Together, these design choices stabilize training and distribute attention more uniformly across instances without auxiliary losses, masking, or multi-stage regularization. We evaluate QG-MIL across six benchmarks spanning whole-slide pathology and cell-level hematology, covering two fundamentally different MIL scales. The best-performing QG-MIL variants outperform leading baselines on all six benchmarks, with an average improvement of +6.1 mean macro F1 points. Attention overlays and attention mass analysis confirm more distributed instance weighting. Ablation studies show that while individual components can match the full model on specific datasets, the QG-MIL design provides the most consistent cross-domain performance and tightest variance when compared to selected baselines. We release a configurable implementation to support reproducibility at: https://github.com/unica-visual-intelligence-lab/QG-MIL
81	arxiv:2606.04405	2026	Low-Rank Decay for Grokking in Scale-Invariant Transformers: A Spectral-Geometric View	Mingyu Li	arXiv	0	A1-arxiv-qknorm;S1-openalex-qknorm	2	SCREEN	exclude [E5] title/abstract: not about attention-logit control or LM training stability	Modern Transformer architectures frequently employ normalization mechanisms such as RMSNorm and Query-Key Normalization, making parts of the model approximately scale-invariant with respect to weight magnitudes. In this regime, standard Frobenius-norm weight decay acts purely along the radial direction of the weight space and cannot directly simplify the function represented by the normalized layer. We study grokking in small algorithmic tasks through this lens and propose \emph{Low-Rank Decay} (LRD), a nuclear-norm-like spectral regularizer whose subgradient -- the polar factor $UV^\top$ -- retains a tangential component even in the scale-invariant setting. This distinction has a concrete dynamical consequence: after the model memorizes the training set and task gradients vanish, L2 decay can no longer reshape the weight spectrum, whereas LRD continues to compress singular values in an $\ell_1$-like fashion. On modular arithmetic tasks, we find that LRD induces rapid effective-rank collapse in Query/Key matrices and expands the data-fraction boundary at which delayed generalization (grokking) occurs. We further provide a spectral-geometric interpretation through the ``needle-to-fan'' expansion of the nuclear-norm subdifferential near low-rank strata.
82	arxiv:2609.12330	2026	Unleashing the Power of Equality Saturation for Tensor Program Superoptimization	Qi Zhan	arXiv	0	A1-arxiv-qknorm;S1-openalex-qknorm	2	SCREEN	exclude [E5] title/abstract: not about attention-logit control or LM training stability	Efficient GPU implementations of tensor programs often require joint optimization of high-level algebraic formulations and low-level execution strategies. However, the resulting search space grows rapidly as transformations combine across operators, making joint optimization difficult to scale. We present EqiForge, a tensor program superoptimizer based on equality saturation. Its unified IR represents high-level tensor expressions and tiled computations in a single expression language. By composing equality rules, EqiForge derives fused implementations such as FlashAttention-style kernels directly from tensor expressions. Early compaction prunes redundant partial programs before completion, while subgraph composition extends the search to larger graphs. Across tensor-program benchmarks, EqiForge achieves a geometric mean speedup of 1.32x and a maximum of 2.74x over the fastest available baseline per configuration. Its attention kernels outperform FlashAttention by up to 1.87x in decode and approach its performance in prefill. EqiForge also discovers new implementations that outperform torch.compile on various Transformer layers, including QK-normalized MLA (3.16x) and mHC (5.84x).
83	doi:10.59431/jda.v5i1.810	2026	Large Language Model Integration in Desktop Applications: A Systematic Literature Review	Deyidi Mokoginta	Journal Dekstop Application (JDA)	0	L1-openalex-llm-for-lit-search;L2-openalex-llm-slr-software-eng	2	SCREEN	exclude [E5] title/abstract: not about attention-logit control or LM training stability	The integration of Large Language Models (LLMs) into desktop applications has accelerated sharply over the past several years, yet no prior systematic review has examined this deployment setting as a distinct research domain — separate from web-based or mobile environments, each carrying different architectural constraints, privacy expectations, and user interaction patterns. This study maps, analyzes, and synthesizes the available scientific evidence on LLM architectures adopted in desktop implementations, the technical barriers arising during integration, the measurable effects on User Experience (UX), and the relative effectiveness of competing deployment strategies. A Systematic Literature Review (SLR) was conducted following PRISMA 2020 reporting guidelines, with research questions structured using the PICOC framework. Systematic searches across seven digital databases — IEEE Xplore, ACM Digital Library, Scopus, Web of Science, Google Scholar, arXiv, and Semantic Scholar — covering 2019–2024 yielded 1,200 initial articles, narrowed to 42 final articles through layered selection and quality appraisal using a modified CASP rubric. GPT-4 and the LLaMA family emerged as the dominant architectures, with model selection driven primarily by parameter scale and privacy requirements. Computational constraints, response latency, and data security ranked as the most serious technical barriers, with Retrieval-Augmented Generation (RAG) and model quantization as the most common mitigations. LLM integration was associated with measurable gains in task efficiency and user satisfaction in 73.8% of reviewed studies, though output predictability and user trust remained persistent challenges. A hybrid deployment approach — combining local models with cloud-based API access — emerged as the most common practice among production implementations. Unlike existing reviews addressing LLM capabilities in general software engineering contexts, this study provides the first systematic map of LLM integration specifically within the desktop application domain, offering concrete guidance for developers working at the intersection of artificial intelligence and desktop software engineering.
84	doi:10.63317/5dnbkz4d44kz	2026	Automatic Generation of Discharge Summaries Using Large Language Models: A Systematic Literature Review	Lucas Molino-Piñar	Language Resources and Evaluation Conference (LREC)	0	L1-openalex-llm-for-lit-search;L2-openalex-llm-slr-software-eng	2	SCREEN	exclude [E5] title/abstract: not about attention-logit control or LM training stability	Discharge summaries are critical documents for continuity of care, yet their manual creation imposes significant burdens on clinical staff.This systematic literature review examines current approaches to automatic generation of discharge summaries using Natural Language Processing (NLP) and Large Language Models (LLMs).Following the Kitchenham guidelines for systematic reviews in software engineering, we searched Scopus and PubMed databases for studies published between 2023 and 2026, identifying 9 primary studies from an initial pool of 102 papers.Our analysis reveals that GPT-4 and its variants dominate current research (appearing in 6 of 9 studies), while open-source alternatives like LLaMA show promise for privacy-preserving deployments.Evaluation primarily relies on automatic metrics (ROUGE, BLEU) combined with human expert assessment.Key challenges include hallucination rates ranging from 33% to 64%, information omission, integration with Electronic Health Record (EHR) systems, and context window limitations.Studies addressing factuality employ human-in-the-loop validation, prompt engineering techniques, and knowledge graph-based correction mechanisms.Despite these challenges, recent implementations demonstrate clinical feasibility, with one study achieving a 94.35% System Usability Score.This review provides a comprehensive synthesis of the state-of-the-art and identifies opportunities for future research in this rapidly evolving field.
85	doi:10.1145/3802827	2026	Enhancing Automated Unit Test Generation with Large Language Models: A Systematic Literature Review	Junwei Zhang	ACM Transactions on Software Engineering and Methodology	4	L1-openalex-llm-for-lit-search;L2-openalex-llm-slr-software-eng	2	SCREEN	exclude [E5] title/abstract: not about attention-logit control or LM training stability	Automated unit test generation is a fundamental yet challenging task in software engineering (SE), playing a critical role in ensuring software correctness, reliability, and maintainability. While traditional approaches such as search-based software testing and symbolic execution have achieved notable success, they often suffer from limited semantic understanding, high configuration costs, and scalability constraints. Recent advances in Large Language Models (LLMs) have fundamentally reshaped the landscape of automated unit testing by enabling models to reason over source code semantics and generate executable, context-aware test cases. Despite the rapid growth of this research area, a comprehensive and task-oriented synthesis of existing work remains lacking. This article presents a systematic literature review of LLM-based unit test generation. This review draws on research from leading SE and AI conferences and journals, including 69 papers published across 25 distinct venues, along with 47 high-quality preprint papers, bringing the total to 116. Our review aims to answer three key research questions: (1) which unit testing tasks have been addressed using LLMs, (2) how LLMs are adapted and integrated into the unit test generation pipeline, and (3) what datasets, benchmarks, and evaluation practices are employed in existing studies. To this end, we organize the literature from a task-centric perspective, covering test generation, test input generation, test oracle generation, and test evolution, and from a methodological perspective, categorizing LLM adaptation strategies into fine-tuning, prompt engineering, and agent-based approaches. Our analysis reveals that current research predominantly focuses on function- and class-level test generation, with comparatively limited attention given to test input generation, oracle construction, and long-term test evolution. Decoder-only LLMs, particularly GPT-family and LLaMA-based models, dominate the field, while encoder-only and encoder–decoder models remain underexplored. We further observe substantial disparities in dataset characteristics, programming language coverage, and evaluation metrics, which hinder fair comparison and reproducibility across studies. Based on empirical evidence extracted from the surveyed literature, we identify key challenges facing LLM-based unit test generation. Building on these findings, we outline several promising research directions, such as dataset optimization, structure-aware context modeling, agent coordination mechanisms, and benchmark enhancement. This review provides a consolidated and evidence-driven foundation for future research, aiming to advance the development of scalable, reliable, and practically applicable LLM-driven unit testing techniques.
86	doi:10.1016/j.jss.2026.113129	2026	Large language models in software project management: A two-phase systematic literature review	Muhammad Saad Amin	Journal of Systems and Software	0	L1-openalex-llm-for-lit-search;L2-openalex-llm-slr-software-eng	2	SCREEN	exclude [E5] title/abstract: not about attention-logit control or LM training stability	Background Large Language Models (LLMs) are increasingly integrated into Software Engineering (SE) processes and practices. However, their application within Software Project Management (SPM) remains fragmented and insufficiently synthesized. This absence of systematic evidence prevents both researchers and practitioners from understanding where LLMs can meaningfully contribute to project management processes in relation to established standards such as the Project Management Body of Knowledge (PMBOK). Objectives This study aims to systematically synthesize existing evidence on the application and integration of LLMs in SPM by means of a two-phase method where Phase I systematically searches for existing secondary studies in the topic of interest, and Phase II (whenever applicable) carries out an SLR in the topic of interest. Method We conducted a two-phase method: Phase I is a tertiary study that used 14 research questions to assess the existence, coverage, and quality of prior secondary studies in LLMs applied to SPM. As no suitable SLR was identified from Phase I, we also conducted Phase II, which employed the SEGRESS guidelines to analyze primary studies that empirically evaluated the use of LLMs in SPM via five research questions. Phase II also assessed coverage across PMBOK processes and synthesized reported mechanisms, outcomes, research designs and evaluation approaches. Results The SLR (Phase II) shows that studies employ encoder-only and decoder-only LLM architectures in equal proportion (17 studies each). Their applications focus on a narrow set of SPM tasks, including effort estimation, project planning, and requirements-related activities. Coverage across PMBOK processes is uneven, with limited attention to areas such as procurement, stakeholder, quality, and cost management. Most studies rely on proof-of-concept evaluations, with scarce industrial validation and heterogeneous evaluation metrics. Conclusion The results indicate that current research on LLMs in SPM provides limited guidance for systematic adoption. One of the main drawbacks for such lack of guidance is the weak empirical validation in many of the included primary studies, limited PMBOK coverage, and lack of context-aware integration of LLMs in SPM. 1
87	doi:10.1109/ieeeconf68869.2026.11461110	2026	Efficient Russian-English Neural Machine Translation with Architectural Optimizations	German Berezin	Systems of Signals Generating and Processing in the Field of on Board Communications	0	S1-openalex-qknorm;snowball-fwd-henry2020	2	SCREEN	exclude [E5] title/abstract: not about attention-logit control or LM training stability	This work addresses efficient neural machine translation deployment for Russian-English translation in resource-constrained onboard communication environments. We propose an enhanced Transformer architecture integrating three key optimizations: Root Mean Square Normalization (RMSNorm) for reduced computational overhead, Query-Key normalization (QKNorm) for robust attention mechanisms, and SwiGLU activation functions for improved parameter efficiency. The approach is evaluated on the OPUS 100 Russian-English parallel corpus using BLEU scores as the primary evaluation metric. The proposed 32-million-parameter model achieves 11.05 BLEU, representing an 8.07 BLEU point improvement over the baseline Transformer with identical parameter count, and demonstrates 3.71x higher parameter efficiency. Ablation studies indicate that architectural design choices significantly impact translation performance within fixed computational budgets. The enhanced architecture enables efficient neural translation deployment on resource-constrained aerospace, maritime, and satellite communication platforms. This work presents a novel architectural combination specifically optimized for Russian-English translation, offering practical insights for deployment of neural translation systems in specialized computing environments.
88	arxiv:2609.34273	2026	Query Expansion and Key Specialization in Transformer Attention Geometry	Vidit Gupta		0	snowball-fwd-henry2020;snowball-fwd-zhai2023	2	SCREEN	exclude [E5] title/abstract: not about attention-logit control or LM training stability	The projection of queries and keys are central to the attention mechanism in Transformer architectures. While they are mathematically symmetric, they play different roles in attention mechanisms. The question of whether there is an effect from their functional distinction on their geometric development in training remains unanswered. We investigate the problem through the training of small GPT-like Transformers on character-level WikiText-103 for three different depths (4, 6, and 8 layers), three types of initialization for queries and keys, and four random seeds, resulting in 36 runs and 54 trajectories of average layers across seeds. We track the effective dimensionality of those layers using participation ratios and discover that effective dimension of queries expand while keys shrink, and that $PR_Q - PR_K$ is positive in all trajectories studied. In connection to attention, the shrinking of keys leads to a narrower spectrum of $QK^\top$ and more peaked attention weights. In order to determine if this connection is causal or coincidental, we directly control the spectrum of keys during training across five seeds: restricting it to make it shrink sharpens the attention with high directional confidence, while keeping it constant to the level of initial dispersion makes attention softer. Additional token-level checkpoint analyses show that the monotonic paired-contrast trend is not universal across pretrained families, but survives as an early-training regime that later decays over a full pretraining run, and the link between interaction-rank geometry and attention entropy remains visible in several models.
89	arxiv:2609.00746	2026	Text Capability Loss in Vision-Language Adaptation: An Attention-Sink Diagnosis	Minsik Choi		0	snowball-fwd-henry2020;snowball-fwd-zhai2023	2	SCREEN	exclude [E5] title/abstract: not about attention-logit control or LM training stability	Fine-tuning a pretrained LLM into a vision-language model (VLM) can erode the backbone's text capability, with the damage concentrated on tasks that require following exact output rules, such as instruction following, chain-of-thought reasoning graded on a strictly parsed final answer, and similar evaluations with strict graders. We trace this gap to attention-sink corruption: VL fine-tuning perturbs the early sink position that anchors a large fraction of attention probability, and how well the base LLM preserves its sink tracks how much of the affected capability survives adaptation. Building on this view, we introduce Sink Strength, a single scalar computed on the base LLM in a few seconds on a single GPU that predicts post-VL degradation without any VL training. It consistently tracks relative degradation across the six VLM-LLM pairs and multiple format-sensitive tasks. Complementing this diagnostic, we find that post-pretraining QK-RMSNorm injection fails to reproduce the protection of native QK-RMSNorm, while several off-the-shelf weight-merging settings fail to recover the lost capability after VL training. These negative results underscore the value of screening backbones with Sink Strength before VL training and narrow the intervention space toward head-selective training-time protection.
90	arxiv:2608.24814	2026	Effective Learning Rate Governs Loss Dynamics in Language Model Pretraining	Zi-Han Liu		5	snowball-fwd-henry2020;snowball-fwd-wortsman2024	2	SCREEN	maybe [BG] effective LR view of loss dynamics	We uncover ELR collapse in language model pretraining: learning rate (LR) and parameter norm govern loss dynamics primarily through their ratio, the effective learning rate (ELR). When ELR is matched across runs, their loss trajectories collapse throughout training despite substantially different LRs and parameter norms. Across optimizers, architectures, datasets, and model scales, mean collapse errors are typically a few x 10^-3, below the seed-to-seed variation measured in a representative configuration. Systematic ablations identify normalization design and the timescale of LR-norm variation as key determinants of collapse precision. Controlled interventions further show that weight decay and Hyperball shape loss dynamics primarily through the ELR schedules they induce. Replacing LR with ELR enables a fitted functional scaling law (FSL) to transfer across norm-control methods. The resulting ELR-based FSL also explains delayed acceleration, a recurring effect of norm control. Together, these results establish ELR as a common coordinate linking LR scheduling, norm control, and loss dynamics.
91	arxiv:2603.13180	2026	MXNorm: Reusing MXFP block scales for efficient tensor normalisation	Callum McLean	arXiv.org	1	snowball-fwd-henry2020;snowball-fwd-wortsman2024	2	SCREEN	exclude [E5] title/abstract: not about attention-logit control or LM training stability	Matrix multiplication performance has long been the major bottleneck to scaling deep learning workloads, which has stimulated the design of new accelerators that use increasingly low-precision number formats. However, improvements in matrix multiplication performance have far outstripped improvements in performance on reductions and elementwise computations, which are still being performed in higher precision. In this work, we propose MXNorm, a drop-in replacement for RMSNorm that estimates the RMS using only the block scales calculated as part of the MXFP8 cast and enables a 32x decrease in the size of reduction needed for normalization. We validate our approximation method on pre-training of Llama 3 models of 125M, 1B and 8B parameters, finding minimal loss of training accuracy compared to a baseline using RMSNorm with MXFP8 matmuls. We also show practical kernel speedups using only torch.compile of up to 2.4x for MXNorm over RMSNorm, corresponding to a 1.3% speedup in Llama 3 8B transformer layers in MXFP8 and a 2.6% speedup in NVFP4.
92	arxiv:2603.05343	2026	Preserving Continuous Symmetry in Discrete Spaces: Geometric-Aware Quantization for SO(3)-Equivariant GNNs	Hao-Yu Zhou	arXiv.org	0	snowball-fwd-henry2020;snowball-fwd-vit22b	2	SCREEN	exclude [E5] title/abstract: not about attention-logit control or LM training stability	Equivariant Graph Neural Networks (GNNs) are essential for physically consistent molecular simulations but suffer from high computational costs and memory bottlenecks, especially with high-order representations. While low-bit quantization offers a solution, applying it naively to rotation-sensitive features destroys the SO(3)-equivariant structure, leading to significant errors and violations of conservation laws. To address this issue, in this work, we propose a Geometric-Aware Quantization (GAQ) framework that compresses and accelerates equivariant models while rigorously preserving continuous symmetry in discrete spaces. Our approach introduces three key contributions: (1) a Magnitude-Direction Decoupled Quantization (MDDQ) scheme that separates invariant lengths from equivariant orientations to maintain geometric fidelity; (2) a symmetry-aware training strategy that treats scalar and vector features with distinct quantization schedules; and (3) a robust attention normalization mechanism to stabilize gradients in low-bit regimes. Experiments on the rMD17 benchmark demonstrate that our W4A8 models match the accuracy of FP32 baselines (9.31 meV vs. 23.20 meV) while reducing Local Equivariance Error (LEE) by over 30x compared to naive quantization. On consumer hardware, GAQ achieves 2.39x inference speedup and 4x memory reduction, enabling stable, energy-conserving molecular dynamics simulations for nanosecond timescales.
93	arxiv:2602.17004	2026	Arcee Trinity Large Technical Report	Varun Singh	arXiv.org	8	snowball-fwd-henry2020;snowball-fwd-wortsman2024	2	SCREEN	exclude [E5] title/abstract: not about attention-logit control or LM training stability	We present the technical report for Arcee Trinity Large, a sparse Mixture-of-Experts model with 400B total parameters and 13B activated per token. Additionally, we report on Trinity Nano and Trinity Mini, with Trinity Nano having 6B total parameters with 1B activated per token, Trinity Mini having 26B total parameters with 3B activated per token. The models'modern architecture includes interleaved local and global attention, gated attention, depth-scaled sandwich norm, and sigmoid routing for Mixture-of-Experts. For Trinity Large, we also introduce a new MoE load balancing strategy titled Soft-clamped Momentum Expert Bias Updates (SMEBU). We train the models using the Muon optimizer. All three models completed training with zero loss spikes. Trinity Nano and Trinity Mini were pre-trained on 10 trillion tokens, and Trinity Large was pre-trained on 17 trillion tokens. The model checkpoints are available at https://huggingface.co/arcee-ai.
94	arxiv:2601.02213	2026	Quantized SO(3)-Equivariant Graph Neural Networks for Efficient Molecular Property Prediction	Hao-Yu Zhou	arXiv.org	0	snowball-fwd-henry2020;snowball-fwd-vit22b	2	SCREEN	exclude [E5] title/abstract: not about attention-logit control or LM training stability	Deploying 3D graph neural networks (GNNs) that are equivariant to 3D rotations (the group SO(3)) on edge devices is challenging due to their high computational cost. This paper addresses the problem by compressing and accelerating an SO(3)-equivariant GNN using low-bit quantization techniques. Specifically, we introduce three innovations for quantized equivariant transformers: (1) a magnitude-direction decoupled quantization scheme that separately quantizes the norm and orientation of equivariant (vector) features, (2) a branch-separated quantization-aware training strategy that treats invariant and equivariant feature channels differently in an attention-based $SO(3)$-GNN, and (3) a robustness-enhancing attention normalization mechanism that stabilizes low-precision attention computations. Experiments on the QM9 and rMD17 molecular benchmarks demonstrate that our 8-bit models achieve accuracy on energy and force predictions comparable to full-precision baselines with markedly improved efficiency. We also conduct ablation studies to quantify the contribution of each component to maintain accuracy and equivariance under quantization, using the Local error of equivariance (LEE) metric. The proposed techniques enable the deployment of symmetry-aware GNNs in practical chemistry applications with 2.37--2.73x faster inference and 4x smaller model size, without sacrificing accuracy or physical symmetry.
95	arxiv:2609.33591	2026	Pretraining Transformers with Quantized Softmax in Attention	Shan-Zhen Zhu		0	snowball-fwd-vit22b;snowball-fwd-wortsman2024	2	SCREEN	exclude [E5] title/abstract: not about attention-logit control or LM training stability	Low-precision Transformer systems increasingly quantize attention matrix multiplications, while softmax often remains at higher precision. During pretraining, an approximate softmax changes the gradients that train the model as well as its forward computation. We study this interaction with K-interval attention, which approximates the exponential using K+1 grid values. We vary per-row grid calibration, interpolation versus hard rounding, and the placement of a straight-through surrogate relative to normalization. We derive the corresponding backward rules, including calibration derivatives, and compare these choices in pretraining experiments matched on model, data, and optimizer. Detaching the row extrema leaves the forward computation unchanged but produces a delayed increase in validation loss. With hard rounding at K=4, min-max calibration and a pre-normalization surrogate incur a large loss gap; changing either choice substantially reduces it. At 124M parameters and 2.5B training tokens, fixed-window calibration with a post-normalization surrogate yields a validation loss gap of +0.019 nats relative to softmax at K=4, and with a pre-normalization surrogate yields +0.004 nats at K=16.
96	arxiv:2608.30320	2026	On the Design of Qwen3.8-Next Architecture: Evaluation, Efficiency, and Training Stability	Zi-Han Qiu		60	snowball-fwd-vit22b;snowball-fwd-wortsman2024	2	SCREEN	maybe [IC] industry architecture study incl. training stability	We describe the architecture and ablations of Qwen3.8-Flash-Next, a sparse mixture-of-experts model with 125B parameters, 6B activated per token, and additional 51B parameters of n-gram embedding tables held off the accelerator. On fourteen pre-training benchmarks the model leads the 397B-A17B predecessor on eight and trails it on the rest by at most 2.6 points, at 1/3 the activated parameters, 1/3 the training tokens, and roughly 1/9 the training FLOPs. Token mixing uses a layer-wise hybrid of Gated DeltaNet (GDN) and global attention, with one full-attention layer in every four; at continued-pretraining time those full-attention layers are replaced by Qwen Sparse Attention (QSA), which scores context at micro-block granularity with a compressed lightweight indexer. The residual stream is widened to four branches and read through an elementwise gate, a design we call the Gated Residual (GR). Capacity is added outside the backbone by a single n-gram embedding layer whose tables are prefetched from host memory. We evaluate every candidate change along three axes: loss together with downstream benchmarks; the cost of the change in training, prefill and decode; and its effect on the optimal hyperparameters and training stability. Loss and downstream accuracy do not always move together: enlarging the n-gram vocabulary lowers loss monotonically while downstream accuracy saturates. The architecture and the Muon optimizer together shift the optimal learning rate and batch size upwards, render batch-size warmup unnecessary, and substantially improve stability under stress tests. Loss, benchmarks, efficiency and stability form one design problem. Solved jointly, they yield a recipe that is simultaneously more efficient, more capable and more stable.
97	arxiv:2603.28743	2026	Rethinking Language Model Scaling under Transferable Hypersphere Optimization	Liliang Ren	arXiv.org	13	snowball-fwd-vit22b;snowball-fwd-wortsman2024	2	SCREEN	exclude [E5] title/abstract: not about attention-logit control or LM training stability	Scaling laws for large language models depend critically on the optimizer and parameterization. Existing hyperparameter transfer laws are mainly developed for first-order optimizers, and they do not structurally prevent training instability at scale. Recent hypersphere optimization methods constrain weight matrices to a fixed-norm hypersphere, offering a promising alternative for more stable scaling. We introduce HyperP (Hypersphere Parameterization), the first framework for transferring optimal learning rates across model width, depth, training tokens, and Mixture-of-Experts (MoE) granularity under the Frobenius-sphere constraint with the Muon optimizer. We prove that weight decay is a first-order no-op on the Frobenius sphere, show that Depth-$\mu$P remains necessary, and find that the optimal learning rate follows the same data-scaling power law with the"magic exponent"0.32 previously observed for AdamW. A single base learning rate tuned at the smallest scale transfers across all compute budgets under HyperP, yielding $1.58\times$ compute efficiency over a strong Muon baseline at $6\times10^{21}$ FLOPs. Moreover, HyperP delivers transferable stability: all monitored instability indicators, including $Z$-values, output RMS, and activation outliers, remain bounded and non-increasing under training FLOPs scaling. We also propose SqrtGate, an MoE gating mechanism derived from the hypersphere constraint that preserves output RMS across MoE granularities for improved granularity scaling, and show that hypersphere optimization enables substantially larger auxiliary load-balancing weights, yielding both strong performance and good expert balance. We release our training codebase at https://github.com/microsoft/ArchScale.
98	arxiv:2602.02400	2026	An Empirical Study on Noisy Data and LLM Pretraining Loss Divergence	Qizhen Zhang	arXiv.org	1	snowball-fwd-vit22b;snowball-fwd-wortsman2024	2	SCREEN	exclude [E5] title/abstract: not about attention-logit control or LM training stability	Large-scale pretraining datasets drive the success of large language models (LLMs). However, these web-scale corpora inevitably contain large amounts of noisy data due to unregulated web content or randomness inherent in data. Although LLM pretrainers often speculate that such noise contributes to instabilities in large-scale LLM pretraining and, in the worst cases, loss divergence, this phenomenon remains poorly understood.In this work, we present a systematic empirical study of whether noisy data causes LLM pretraining divergences and how it does so. By injecting controlled synthetic uniformly random noise into otherwise clean datasets, we analyze training dynamics across model sizes ranging from 480M to 5.2B parameters. We show that noisy data indeed induces training loss divergence, and that the probability of divergence depends strongly on the noise type, amount of noise, and model scale. We further find that noise-induced divergences exhibit activation patterns distinct from those caused by high learning rates, and we provide diagnostics that differentiate these two failure modes. Together, these results provide a large-scale, controlled characterization of how noisy data affects loss divergence in LLM pretraining.
99	arxiv:2601.02031	2026	Output Embedding Centering for Stable LLM Pretraining	Felix Stollenwerk	arXiv.org	4	snowball-fwd-vit22b;snowball-fwd-wortsman2024	2	SCREEN	maybe [IA] output-logit divergence; z-loss / softcap context	Pretraining of large language models is not only expensive but also prone to certain training instabilities. A specific instability that often occurs at the end of training is output logit divergence. The most widely used mitigation strategies, z-loss and logit soft-capping, merely address the symptoms rather than the underlying cause of the problem. In this paper, we analyze the instability from the perspective of the output embeddings'geometry and identify anisotropic embeddings as its source. Based on this, we propose output embedding centering (OEC) as a new mitigation strategy, and demonstrate that it suppresses output logit divergence. OEC can be implemented in two different ways: as a deterministic operation called $\mu$-centering, or a regularization method called $\mu$-loss. Our experiments show that both variants outperform z-loss in terms of training stability, while being on par with logit soft-capping. This holds true both in the presence and the absence of weight tying. As a secondary result, we find that $\mu$-loss is significantly less sensitive to regularization hyperparameter tuning than z-loss.
100	title:intriguingpropertiesoftransformertraininginstabilities2023		Intriguing Properties of Transformer Training Instabilities, 2023		To appear	0	snowball-back-vit22b	1	SCREEN	exclude [E4] earlier title / duplicate of Wortsman et al. 2309.14322
101	title:intriguingpropertiesoftransformertraininginstabilities		Intriguing properties of transformer training instabilities		To appear	0	snowball-back-wortsman2024	1	SCREEN	exclude [E4] earlier title / duplicate of Wortsman et al. 2309.14322
102	title:2022resultstoprobeforentropycollapsephenomenonalllaterexperimentssection43andappendixfwithdeeptransformersanddeepnormaredonebytatianalikhomanenko		2022) results to probe for entropy collapse phenomenon. All later experiments, Section 4.3 and Appendix F, with deep transformers and deepnorm are done by Tatiana Likhomanenko			0	snowball-back-zhai2023	1	SCREEN	exclude [E4] citation-context fragment, not a paper
103	title:flashnormfastnormalizationforllms		FlashNorm: fast normalization for LLMs	Nils Graef		1	snowball-fwd-henry2020	1	SCREEN	exclude [E5] title/abstract: not about attention-logit control or LM training stability
104	arxiv:2005.09561	2020	Normalized Attention Without Probability Cage	Oliver Richter	arXiv.org	29	snowball-back-henry2020	1	SCREEN	exclude [E5] title/abstract: not about attention-logit control or LM training stability	Attention architectures are widely used; they recently gained renewed popularity with Transformers yielding a streak of state of the art results. Yet, the geometrical implications of softmax-attention remain largely unexplored. In this work we highlight the limitations of constraining attention weights to the probability simplex and the resulting convex hull of value vectors. We show that Transformers are sequence length dependent biased towards token isolation at initialization and contrast Transformers to simple max- and sum-pooling - two strong baselines rarely reported. We propose to replace the softmax in self-attention with normalization, yielding a hyperparameter and data-bias robust, generally applicable architecture. We support our insights with empirical results from more than 25,000 trained models. All results and implementations are made available.
105	title:powernormrethinkingbatchnormalizationintransformers	2020	PowerNorm: Rethinking Batch Normalization in Transformers	Sheng Shen	International Conference on Machine Learning	95	snowball-back-henry2020	1	SCREEN	exclude [E5] title/abstract: not about attention-logit control or LM training stability
106	arxiv:2002.04745	2020	On Layer Normalization in the Transformer Architecture	Ruibin Xiong	International Conference on Machine Learning	1694	snowball-back-henry2020	1	SCREEN	include [BG] Pre-LN vs Post-LN, warmup and LR (2020)	The Transformer is widely used in natural language processing tasks. To train a Transformer however, one usually needs a carefully designed learning rate warm-up stage, which is shown to be crucial to the final performance but will slow down the optimization and bring more hyperparameter tunings. In this paper, we first study theoretically why the learning rate warm-up stage is essential and show that the location of layer normalization matters. Specifically, we prove with mean field theory that at initialization, for the original-designed Post-LN Transformer, which places the layer normalization between the residual blocks, the expected gradients of the parameters near the output layer are large. Therefore, using a large learning rate on those gradients makes the training unstable. The warm-up stage is practically helpful for avoiding this problem. On the other hand, our theory also shows that if the layer normalization is put inside the residual blocks (recently proposed as Pre-LN Transformer), the gradients are well-behaved at initialization. This motivates us to remove the warm-up stage for the training of Pre-LN Transformers. We show in our experiments that Pre-LN Transformers without the warm-up stage can reach comparable results with baselines while requiring significantly less training time and hyper-parameter tuning on a wide range of applications.
107	arxiv:2101.03961	2021	Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity	W. Fedus	Journal of machine learning research	5199	snowball-back-vit22b	1	SCREEN	exclude [E5] title/abstract: not about attention-logit control or LM training stability	In deep learning, models typically reuse the same parameters for all inputs. Mixture of Experts (MoE) defies this and instead selects different parameters for each incoming example. The result is a sparsely-activated model -- with outrageous numbers of parameters -- but a constant computational cost. However, despite several notable successes of MoE, widespread adoption has been hindered by complexity, communication costs and training instability -- we address these with the Switch Transformer. We simplify the MoE routing algorithm and design intuitive improved models with reduced communication and computational costs. Our proposed training techniques help wrangle the instabilities and we show large sparse models may be trained, for the first time, with lower precision (bfloat16) formats. We design models based off T5-Base and T5-Large to obtain up to 7x increases in pre-training speed with the same computational resources. These improvements extend into multilingual settings where we measure gains over the mT5-Base version across all 101 languages. Finally, we advance the current scale of language models by pre-training up to trillion parameter models on the"Colossal Clean Crawled Corpus"and achieve a 4x speedup over the T5-XXL model.
108	arxiv:2110.09456	2021	NormFormer: Improved Transformer Pretraining with Extra Normalization	Sam Shleifer	arXiv.org	102	snowball-back-zhai2023	1	SCREEN	include [IA] NormFormer: extra normalisation	During pretraining, the Pre-LayerNorm transformer suffers from a gradient magnitude mismatch: gradients at early layers are much larger than at later layers. These issues can be alleviated by our proposed NormFormer architecture, which adds three normalization operations to each layer: a Layer Norm after self attention, head-wise scaling of self-attention outputs, and a Layer Norm after the first fully connected layer. The extra operations incur negligible compute cost (+0.4% parameter increase), but improve pretraining perplexity and downstream task performance for both causal and masked language models ranging from 125 Million to 2.7 Billion parameters. For example, adding NormFormer on top of our strongest 1.3B parameter baseline can reach equal perplexity 24% faster, or converge 0.27 perplexity better in the same compute budget. This model reaches GPT3-Large (1.3B) zero shot performance 60% faster. For masked language modeling, NormFormer improves fine-tuned GLUE performance by 1.9% on average. Code to train NormFormer models is available in fairseq https://github.com/pytorch/fairseq/tree/main/examples/normformer .
109	arxiv:2208.07339	2022	LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale	Tim Dettmers	Advances in Neural Information Processing Systems 35	1380	snowball-back-wortsman2024	1	SCREEN	exclude [E5] title/abstract: not about attention-logit control or LM training stability	Large language models have been widely adopted but require significant GPU memory for inference. We develop a procedure for Int8 matrix multiplication for feed-forward and attention projection layers in transformers, which cut the memory needed for inference by half while retaining full precision performance. With our method, a 175B parameter 16/32-bit checkpoint can be loaded, converted to Int8, and used immediately without performance degradation. This is made possible by understanding and working around properties of highly systematic emergent features in transformer language models that dominate attention and transformer predictive performance. To cope with these features, we develop a two-part quantization procedure, LLM.int8(). We first use vector-wise quantization with separate normalization constants for each inner product in the matrix multiplication, to quantize most of the features. However, for the emergent outliers, we also include a new mixed-precision decomposition scheme, which isolates the outlier feature dimensions into a 16-bit matrix multiplication while still more than 99.9% of values are multiplied in 8-bit. Using LLM.int8(), we show empirically it is possible to perform inference in LLMs with up to 175B parameters without any performance degradation. This result makes such models much more accessible, for example making it possible to use OPT-175B/BLOOM on a single server with consumer GPUs. We open-source our software.
110	arxiv:2203.00555	2022	DeepNet: Scaling Transformers to 1,000 Layers	Hongyu Wang	IEEE Transactions on Pattern Analysis and Machine Intelligence	284	snowball-back-zhai2023	1	SCREEN	exclude [E5] title/abstract: not about attention-logit control or LM training stability	In this paper, we propose a simple yet effective method to stabilize extremely deep Transformers. Specifically, we introduce a new normalization function (DeepNorm) to modify the residual connection in Transformer, accompanying with theoretically derived initialization. In-depth theoretical analysis shows that model updates can be bounded in a stable way. The proposed method combines the best of two worlds, i.e., good performance of Post-LN and stable training of Pre-LN, making DeepNorm a preferred alternative. We successfully scale Transformers up to 1,000 layers (i.e., 2,500 attention and feed-forward network sublayers) without difficulty, which is one order of magnitude deeper than previous deep Transformers. Extensive experiments demonstrate that DeepNet has superior performance across various benchmarks, including machine translation, language modeling (i.e., BERT, GPT) and vision pre-training (i.e., BEiT). Remarkably, on a multilingual benchmark with 7,482 translation directions, our 200-layer model with 3.2B parameters significantly outperforms the 48-layer state-of-the-art model with 12B parameters by 5 BLEU points, which indicates a promising scaling direction.
111	title:canrecoverfromattentionentropycollapsehappensinencoderlayersandnicelyconverge100l100lsuffersfromitandcandiverge	2022	can recover from attention entropy collapse (happens in encoder layers) and nicely converge, 100L-100L suffers from it and can diverge			0	snowball-back-zhai2023	1	SCREEN	exclude [E4] citation-context fragment, not a paper
112	title:wealsoobservedthesametrendofdecreasingmodelperformancewithincreasingthemodeldepthattentionentropyisnicelyboundedacrossalldepthssimilarlytoasr11	2022	We also observed the same trend of decreasing model performance with increasing the model depth. Attention entropy is nicely bounded across all depths similarly to ASR11			0	snowball-back-zhai2023	1	SCREEN	exclude [E4] citation-context fragment, not a paper
113	arxiv:2202.09785	2022	DualSC: Automatic Generation and Summarization of Shellcode via Transformer and Dual Learning	Guang Yang	IEEE International Conference on Software Analysis, Evolution, and Reengineering	35	snowball-fwd-henry2020	1	SCREEN	exclude [E5] title/abstract: not about attention-logit control or LM training stability	A shellcode is a small piece of code and it is executed to exploit a software vulnerability, which allows the target computer to execute arbitrary commands from the attacker through a code injection attack. Similar to the purpose of automated vulnerability generation techniques, the automated generation of shellcode can generate attack instructions, which can be used to detect vulnerabilities and implement defensive measures. While the automated summarization of shellcode can help users unfamiliar with shellcode and network information security understand the intent of shellcode attacks. In this study, we propose a novel approach DualSC to solve the automatic shellcode generation and summarization tasks. Specifically, we formalize automatic shellcode generation and summarization as dual tasks, use a shallow Transformer for model construction, and design a normalization method Adjust_QKNorm to adapt these low-resource tasks (i.e., insufficient training data). Finally, to alleviate the out-of-vocabulary problem, we propose a rule-based repair component to improve the performance of automatic shellcode generation. In our empirical study, we select a high-quality corpus Shellcode_IA32 as our empirical subject. This corpus was gathered from two real-world projects based on the line-by-line granularity. We first compare DualSC with six state-of-the-art baselines from the code generation and code summarization domains in terms of four performance measures. The comparison results show the competitiveness of DualSC. Then, we verify the effectiveness of the component setting in DualSC. Finally, we conduct a human study to further verify the effectiveness of DualSC.
114	doi:10.1101/2023.08.07.23293769	2023	Large Language Models (LLMs) and Empathy – A Systematic Review	Vera Sorin	medRxiv	45	L1-openalex-llm-for-lit-search	1	SCREEN	exclude [E6] uses LLMs as a research subject, not as a review tool	Abstract Purpose Empathy, a cornerstone of human interaction, is a unique quality to humans that Large Language Models (LLMs) are believed to lack. Our study aims to review the literature on the capacity of LLMs in demonstrating empathy Methods We conducted a literature search on MEDLINE up to July 2023. Seven publications ultimately met the inclusion criteria. Results All studies included in this review were published in 2023. All studies but one focused on ChatGPT-3.5 by OpenAI. Only one study evaluated empathy based on objective metrics, and all others used subjective human assessment. The studies reported LLMs to exhibits elements of empathy, including emotions recognition and providing emotionally supportive responses in diverse contexts, most of which were related to healthcare. In some cases, LLMs were observed to outperform humans in empathy-related tasks. Conclusion LLMs demonstrated some aspects of empathy in variable scenarios, mainly related to healthcare. The empathy may be considered “cognitive” empathy. Social skills are a fundamental aspect of intelligence, thus further research is imperative to enhance these skills in AI.
115	doi:10.1016/j.jval.2023.09.030	2023	P21 A Comparative Analysis of Large Language Models (LLM) Utilised in Systematic Literature Review	H Rathi	Value in Health	4	L1-openalex-llm-for-lit-search	1	SCREEN	exclude [E6] uses LLMs as a research subject, not as a review tool
116	doi:10.1145/3539618.3591703	2023	Can ChatGPT Write a Good Boolean Query for Systematic Review Literature Search?	Shuai Wang	International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR)	205	L1-openalex-llm-for-lit-search	1	SCREEN	include [LLM] ChatGPT writing Boolean search queries (SIGIR)	Systematic reviews are comprehensive literature reviews for a highly focused research question. These reviews are considered the highest form of evidence in medicine. Complex Boolean queries are developed as part of the systematic review creation process to retrieve literature, as they permit reproducibility and understandability. However, it is difficult and time-consuming to develop high-quality Boolean queries, often requiring the expertise of expert searchers like librarians. Recent advances in transformer-based generative models have shown their ability to effectively follow user instructions and generate answers based on these instructions. In this paper, we investigate ChatGPT as a means for automatically formulating and refining complex Boolean queries for systematic review literature search. Overall, our research finds that ChatGPT has the potential to generate effective Boolean queries. The ability of ChatGPT to follow complex instructions and generate highly precise queries makes it a tool of potential value for researchers conducting systematic reviews, particularly for rapid reviews where time is a constraint and where one can trade off higher precision for lower recall. We also identify several caveats in using ChatGPT for this task, highlighting that this technology needs further validation before it is suitable for widespread uptake.
117	doi:10.2196/48996	2023	Automated Paper Screening for Clinical Reviews Using Large Language Models: Data Analysis Study	Eddie Guo	Journal of Medical Internet Research	205	L1-openalex-llm-for-lit-search	1	SCREEN	maybe [LLM] automated screening with LLMs (clinical)	BACKGROUND: The systematic review of clinical research papers is a labor-intensive and time-consuming process that often involves the screening of thousands of titles and abstracts. The accuracy and efficiency of this process are critical for the quality of the review and subsequent health care decisions. Traditional methods rely heavily on human reviewers, often requiring a significant investment of time and resources. OBJECTIVE: This study aims to assess the performance of the OpenAI generative pretrained transformer (GPT) and GPT-4 application programming interfaces (APIs) in accurately and efficiently identifying relevant titles and abstracts from real-world clinical review data sets and comparing their performance against ground truth labeling by 2 independent human reviewers. METHODS: We introduce a novel workflow using the Chat GPT and GPT-4 APIs for screening titles and abstracts in clinical reviews. A Python script was created to make calls to the API with the screening criteria in natural language and a corpus of title and abstract data sets filtered by a minimum of 2 human reviewers. We compared the performance of our model against human-reviewed papers across 6 review papers, screening over 24,000 titles and abstracts. RESULTS: -score of 0.60, a sensitivity of excluded papers of 0.91, and a sensitivity of included papers of 0.76. The interrater variability between 2 independent human screeners was κ=0.46, and the prevalence and bias-adjusted κ between our proposed methods and the consensus-based human decisions was κ=0.96. On a randomly selected subset of papers, the GPT models demonstrated the ability to provide reasoning for their decisions and corrected their initial decisions upon being asked to explain their reasoning for incorrect classifications. CONCLUSIONS: Large language models have the potential to streamline the clinical review process, save valuable time and effort for researchers, and contribute to the overall quality of clinical reviews. By prioritizing the workflow and acting as an aid rather than a replacement for researchers and reviewers, models such as GPT-4 can enhance efficiency and lead to more accurate and reliable conclusions in medical research.
118	doi:10.1101/2023.09.03.23294842	2023	How Large Language Models Perform on the United States Medical Licensing Examination: A Systematic Review	Dana Brin	medRxiv	16	L1-openalex-llm-for-lit-search	1	SCREEN	exclude [E6] uses LLMs as a research subject, not as a review tool	ABSTRACT Objective The United States Medical Licensing Examination (USMLE) assesses physicians’ competency and passing is a requirement to practice medicine in the U.S. With the emergence of large language models (LLMs) like ChatGPT and GPT-4, understanding their performance on these exams illuminates their potential in medical education and healthcare. Materials and Methods A literature search following the 2020 PRISMA guidelines was conducted, focusing on studies using official USMLE questions and publicly available LLMs. Results Three relevant studies were found, with GPT-4 showcasing the highest accuracy rates of 80-90% on the USMLE. Open-ended prompts typically outperformed multiple-choice ones, with 5-shot prompting slightly edging out zero-shot. Conclusion LLMs, especially GPT-4, display proficiency in tackling USMLE-standard questions. While the USMLE is a structured evaluation tool, it may not fully capture the expansive capabilities and limitations of LLMs in medical scenarios. As AI integrates further into healthcare, ongoing assessments against trusted benchmarks are essential.
119	arxiv:2308.06610	2023	Bio-SIEVE: Exploring Instruction Tuning Large Language Models for Systematic Review Automation	Ambrose Robinson	arXiv (Cornell University)	9	L1-openalex-llm-for-lit-search	1	SCREEN	exclude [E6] uses LLMs as a research subject, not as a review tool	Medical systematic reviews can be very costly and resource intensive. We explore how Large Language Models (LLMs) can support and be trained to perform literature screening when provided with a detailed set of selection criteria. Specifically, we instruction tune LLaMA and Guanaco models to perform abstract screening for medical systematic reviews. Our best model, Bio-SIEVE, outperforms both ChatGPT and trained traditional approaches, and generalises better across medical domains. However, there remains the challenge of adapting the model to safety-first scenarios. We also explore the impact of multi-task training with Bio-SIEVE-Multi, including tasks such as PICO extraction and exclusion reasoning, but find that it is unable to match single-task Bio-SIEVE's performance. We see Bio-SIEVE as an important step towards specialising LLMs for the biomedical systematic review process and explore its future developmental opportunities. We release our models, code and a list of DOIs to reconstruct our dataset for reproducibility.
120	arxiv:2309.17453	2023	Efficient Streaming Language Models with Attention Sinks	Guang-Xuan Xiao	International Conference on Learning Representations	2726	snowball-back-imu1	1	SCREEN	maybe [IM] attention sinks (inference paper, mechanism background)	Deploying Large Language Models (LLMs) in streaming applications such as multi-round dialogue, where long interactions are expected, is urgently needed but poses two major challenges. Firstly, during the decoding stage, caching previous tokens' Key and Value states (KV) consumes extensive memory. Secondly, popular LLMs cannot generalize to longer texts than the training sequence length. Window attention, where only the most recent KVs are cached, is a natural approach -- but we show that it fails when the text length surpasses the cache size. We observe an interesting phenomenon, namely attention sink, that keeping the KV of initial tokens will largely recover the performance of window attention. In this paper, we first demonstrate that the emergence of attention sink is due to the strong attention scores towards initial tokens as a"sink"even if they are not semantically important. Based on the above analysis, we introduce StreamingLLM, an efficient framework that enables LLMs trained with a finite length attention window to generalize to infinite sequence lengths without any fine-tuning. We show that StreamingLLM can enable Llama-2, MPT, Falcon, and Pythia to perform stable and efficient language modeling with up to 4 million tokens and more. In addition, we discover that adding a placeholder token as a dedicated attention sink during pre-training can further improve streaming deployment. In streaming settings, StreamingLLM outperforms the sliding window recomputation baseline by up to 22.2x speedup. Code and datasets are provided at https://github.com/mit-han-lab/streaming-llm.
121	arxiv:2312.14820	2023	How Smooth Is Attention?	Valérie Castin	International Conference on Machine Learning	53	snowball-fwd-vit22b	1	SCREEN	exclude [E5] title/abstract: not about attention-logit control or LM training stability	Self-attention and masked self-attention are at the heart of Transformers' outstanding success. Still, our mathematical understanding of attention, in particular of its Lipschitz properties - which are key when it comes to analyzing robustness and expressive power - is incomplete. We provide a detailed study of the Lipschitz constant of self-attention in several practical scenarios, discussing the impact of the sequence length $n$ and layer normalization on the local Lipschitz constant of both unmasked and masked self-attention. In particular, we show that for inputs of length $n$ in any compact set, the Lipschitz constant of self-attention is bounded by $\sqrt{n}$ up to a constant factor and that this bound is tight for reasonable sequence lengths. When the sequence length $n$ is too large for the previous bound to be tight, which we refer to as the mean-field regime, we provide an upper bound and a matching lower bound which are independent of $n$. Our mean-field framework for masked self-attention is novel and of independent interest. Our experiments on pretrained and randomly initialized BERT and GPT-2 support our theoretical findings.
122	arxiv:2311.01906	2023	Simplifying Transformer Blocks	Bobby He	International Conference on Learning Representations	62	snowball-fwd-vit22b	1	SCREEN	exclude [E5] title/abstract: not about attention-logit control or LM training stability	A simple design recipe for deep Transformers is to compose identical building blocks. But standard transformer blocks are far from simple, interweaving attention and MLP sub-blocks with skip connections&normalisation layers in precise arrangements. This complexity leads to brittle architectures, where seemingly minor changes can significantly reduce training speed, or render models untrainable. In this work, we ask to what extent the standard transformer block can be simplified? Combining signal propagation theory and empirical observations, we motivate modifications that allow many block components to be removed with no loss of training speed, including skip connections, projection or value parameters, sequential sub-blocks and normalisation layers. In experiments on both autoregressive decoder-only and BERT encoder-only models, our simplified transformers emulate the per-update training speed and performance of standard transformers, while enjoying 15% faster training throughput, and using 15% fewer parameters.
123	arxiv:2510.17189	2023	SOLE: Hardware-Software Co-design of Softmax and LayerNorm for Efficient Transformer Inference	Wen-Xun Wang	2023 IEEE/ACM International Conference on Computer Aided Design (ICCAD)	57	snowball-fwd-vit22b	1	SCREEN	exclude [E5] title/abstract: not about attention-logit control or LM training stability	Transformers have shown remarkable performance in both natural language processing (NLP) and computer vision (CV) tasks. However, their real-time inference speed and efficiency are limited due to the inefficiency in Softmax and Layer Normalization (LayerNorm). Previous works based on function approximation suffer from inefficient implementation as they place emphasis on computation while disregarding memory overhead concerns. Moreover, such methods rely on retraining to compensate for approximation error which can be costly and inconvenient. In this paper, we present SOLE, a hardware-software co-design for Softmax and LayerNorm which is composed of E2Softmax and AILayerNorm. E2Softmax utilizes log2 quantization of exponent function and log-based division to approximate Softmax while AILayerNorm adopts low-precision statistic calculation. Compared with state-of-the-art designs, we achieve both low-precision calculation and low bit-width storage on Softmax and LayerNorm. Experiments show that SOLE maintains inference accuracy without retraining while offering orders of magnitude speedup and energy savings over GPU, achieving 3.04×, 3.86× energy-efficiency improvements and 2.82×, 3.32× area-efficiency improvements over prior state-of-the-art custom hardware for Softmax and LayerNorm, respectively.
124	arxiv:2310.02994	2023	Multiple Physics Pretraining for Physical Surrogate Models	Michael McCabe	arXiv.org	122	snowball-fwd-vit22b	1	SCREEN	exclude [E5] title/abstract: not about attention-logit control or LM training stability	We introduce multiple physics pretraining (MPP), an autoregressive task-agnostic pretraining approach for physical surrogate modeling of spatiotemporal systems with transformers. In MPP, rather than training one model on a specific physical system, we train a backbone model to predict the dynamics of multiple heterogeneous physical systems simultaneously in order to learn features that are broadly useful across systems and facilitate transfer. In order to learn effectively in this setting, we introduce a shared embedding and normalization strategy that projects the fields of multiple systems into a shared embedding space. We validate the efficacy of our approach on both pretraining and downstream tasks over a broad fluid mechanics-oriented benchmark. We show that a single MPP-pretrained transformer is able to match or outperform task-specific baselines on all pretraining sub-tasks without the need for finetuning. For downstream tasks, we demonstrate that finetuning MPP-trained models results in more accurate predictions across multiple time-steps on systems with previously unseen physical components or higher dimensional systems compared to training from scratch or finetuning pretrained video foundation models. We open-source our code and model weights trained at multiple scales for reproducibility.
125	arxiv:2306.12059	2023	EquiformerV2: Improved Equivariant Transformer for Scaling to Higher-Degree Representations	Yidong Liao	International Conference on Learning Representations	407	snowball-fwd-vit22b	1	SCREEN	exclude [E5] title/abstract: not about attention-logit control or LM training stability	Equivariant Transformers such as Equiformer have demonstrated the efficacy of applying Transformers to the domain of 3D atomistic systems. However, they are limited to small degrees of equivariant representations due to their computational complexity. In this paper, we investigate whether these architectures can scale well to higher degrees. Starting from Equiformer, we first replace $SO(3)$ convolutions with eSCN convolutions to efficiently incorporate higher-degree tensors. Then, to better leverage the power of higher degrees, we propose three architectural improvements -- attention re-normalization, separable $S^2$ activation and separable layer normalization. Putting this all together, we propose EquiformerV2, which outperforms previous state-of-the-art methods on large-scale OC20 dataset by up to $9\%$ on forces, $4\%$ on energies, offers better speed-accuracy trade-offs, and $2\times$ reduction in DFT calculations needed for computing adsorption energies. Additionally, EquiformerV2 trained on only OC22 dataset outperforms GemNet-OC trained on both OC20 and OC22 datasets, achieving much better data efficiency. Finally, we compare EquiformerV2 with Equiformer on QM9 and OC20 S2EF-2M datasets to better understand the performance gain brought by higher degrees.
126	arxiv:2305.14858	2023	Pre-RMSNorm and Pre-CRMSNorm Transformers: Equivalent and Efficient Pre-LN Transformers	Zi-Xuan Jiang	Neural Information Processing Systems	45	snowball-fwd-vit22b	1	SCREEN	exclude [E5] title/abstract: not about attention-logit control or LM training stability	Transformers have achieved great success in machine learning applications. Normalization techniques, such as Layer Normalization (LayerNorm, LN) and Root Mean Square Normalization (RMSNorm), play a critical role in accelerating and stabilizing the training of Transformers. While LayerNorm recenters and rescales input vectors, RMSNorm only rescales the vectors by their RMS value. Despite being more computationally efficient, RMSNorm may compromise the representation ability of Transformers. There is currently no consensus regarding the preferred normalization technique, as some models employ LayerNorm while others utilize RMSNorm, especially in recent large language models. It is challenging to convert Transformers with one normalization to the other type. While there is an ongoing disagreement between the two normalization types, we propose a solution to unify two mainstream Transformer architectures, Pre-LN and Pre-RMSNorm Transformers. By removing the inherent redundant mean information in the main branch of Pre-LN Transformers, we can reduce LayerNorm to RMSNorm, achieving higher efficiency. We further propose the Compressed RMSNorm (CRMSNorm) and Pre-CRMSNorm Transformer based on a lossless compression of the zero-mean vectors. We formally establish the equivalence of Pre-LN, Pre-RMSNorm, and Pre-CRMSNorm Transformer variants in both training and inference. It implies that Pre-LN Transformers can be substituted with Pre-(C)RMSNorm counterparts at almost no cost, offering the same arithmetic functionality along with free efficiency improvement. Experiments demonstrate that we can reduce the training and inference time of Pre-LN Transformers by 1% - 10%.
127	arxiv:2304.03208	2023	Cerebras-GPT: Open Compute-Optimal Language Models Trained on the Cerebras Wafer-Scale Cluster	Nolan Dey	arXiv.org	148	snowball-fwd-vit22b	1	SCREEN	maybe [BG] muP in small open models	We study recent research advances that improve large language models through efficient pre-training and scaling, and open datasets and tools. We combine these advances to introduce Cerebras-GPT, a family of open compute-optimal language models scaled from 111M to 13B parameters. We train Cerebras-GPT models on the Eleuther Pile dataset following DeepMind Chinchilla scaling rules for efficient pre-training (highest accuracy for a given compute budget). We characterize the predictable power-law scaling and compare Cerebras-GPT with other publicly-available models to show all Cerebras-GPT models have state-of-the-art training efficiency on both pre-training and downstream objectives. We describe our learnings including how Maximal Update Parameterization ($\mu$P) can further improve large model scaling, improving accuracy and hyperparameter predictability at scale. We release our pre-trained models and code, making this paper the first open and reproducible work comparing compute-optimal model scaling to models trained on fixed dataset sizes. Cerebras-GPT models are available on HuggingFace: https://huggingface.co/cerebras.
128	arxiv:2410.23749	2024	LATST: Are Transformers Necessarily Complex for Time-Series Forecasting	Dizhen Liang	arXiv	0	A2-arxiv-logit-instability	1	SCREEN	exclude [E5] title/abstract: not about attention-logit control or LM training stability	Transformer-based architectures have achieved remarkable success in natural language processing and computer vision. However, their performance in multivariate long-term forecasting often falls short compared to simpler linear baselines. Previous research has identified the traditional attention mechanism as a key factor limiting their effectiveness in this domain. To bridge this gap, we introduce LATST, a novel approach designed to mitigate entropy collapse and training instability common challenges in Transformer-based time series forecasting. We rigorously evaluate LATST across multiple real-world multivariate time series datasets, demonstrating its ability to outperform existing state-of-the-art Transformer models. Notably, LATST manages to achieve competitive performance with fewer parameters than some linear models on certain datasets, highlighting its efficiency and effectiveness.
129	doi:10.1038/s41746-024-01157-x	2024	The ethics of ChatGPT in medicine and healthcare: a systematic review on Large Language Models (LLMs)	Joschka Haltaufderheide	npj Digital Medicine	393	L1-openalex-llm-for-lit-search	1	SCREEN	exclude [E6] uses LLMs as a research subject, not as a review tool	With the introduction of ChatGPT, Large Language Models (LLMs) have received enormous attention in healthcare. Despite potential benefits, researchers have underscored various ethical implications. While individual instances have garnered attention, a systematic and comprehensive overview of practical applications currently researched and ethical issues connected to them is lacking. Against this background, this work maps the ethical landscape surrounding the current deployment of LLMs in medicine and healthcare through a systematic review. Electronic databases and preprint servers were queried using a comprehensive search strategy which generated 796 records. Studies were screened and extracted following a modified rapid review approach. Methodological quality was assessed using a hybrid approach. For 53 records, a meta-aggregative synthesis was performed. Four general fields of applications emerged showcasing a dynamic exploration phase. Advantages of using LLMs are attributed to their capacity in data analysis, information provisioning, support in decision-making or mitigating information loss and enhancing information accessibility. However, our study also identifies recurrent ethical concerns connected to fairness, bias, non-maleficence, transparency, and privacy. A distinctive concern is the tendency to produce harmful or convincing but inaccurate content. Calls for ethical guidance and human oversight are recurrent. We suggest that the ethical guidance debate should be reframed to focus on defining what constitutes acceptable human oversight across the spectrum of applications. This involves considering the diversity of settings, varying potentials for harm, and different acceptable thresholds for performance and certainty in healthcare. Additionally, critical inquiry is needed to evaluate the necessity and justification of LLMs' current experimental use.
130	doi:10.2196/52597	2024	Large Language Models and Empathy: Systematic Review	Vera Sorin	Journal of Medical Internet Research	240	L1-openalex-llm-for-lit-search	1	SCREEN	exclude [E6] uses LLMs as a research subject, not as a review tool	BACKGROUND: Empathy, a fundamental aspect of human interaction, is characterized as the ability to experience another being's emotions within oneself. In health care, empathy is a fundamental for health care professionals and patients' interaction. It is a unique quality to humans that large language models (LLMs) are believed to lack. OBJECTIVE: We aimed to review the literature on the capacity of LLMs in demonstrating empathy. METHODS: We conducted a literature search on MEDLINE, Google Scholar, PsyArXiv, medRxiv, and arXiv between December 2022 and February 2024. We included English-language full-length publications that evaluated empathy in LLMs' outputs. We excluded papers evaluating other topics related to emotional intelligence that were not specifically empathy. The included studies' results, including the LLMs used, performance in empathy tasks, and limitations of the models, along with studies' metadata were summarized. RESULTS: A total of 12 studies published in 2023 met the inclusion criteria. ChatGPT-3.5 (OpenAI) was evaluated in all studies, with 6 studies comparing it with other LLMs such GPT-4, LLaMA (Meta), and fine-tuned chatbots. Seven studies focused on empathy within a medical context. The studies reported LLMs to exhibit elements of empathy, including emotions recognition and emotional support in diverse contexts. Evaluation metric included automatic metrics such as Recall-Oriented Understudy for Gisting Evaluation and Bilingual Evaluation Understudy, and human subjective evaluation. Some studies compared performance on empathy with humans, while others compared between different models. In some cases, LLMs were observed to outperform humans in empathy-related tasks. For example, ChatGPT-3.5 was evaluated for its responses to patients' questions from social media, where ChatGPT's responses were preferred over those of humans in 78.6% of cases. Other studies used subjective readers' assigned scores. One study reported a mean empathy score of 1.84-1.9 (scale 0-2) for their fine-tuned LLM, while a different study evaluating ChatGPT-based chatbots reported a mean human rating of 3.43 out of 4 for empathetic responses. Other evaluations were based on the level of the emotional awareness scale, which was reported to be higher for ChatGPT-3.5 than for humans. Another study evaluated ChatGPT and GPT-4 on soft-skills questions in the United States Medical Licensing Examination, where GPT-4 answered 90% of questions correctly. Limitations were noted, including repetitive use of empathic phrases, difficulty following initial instructions, overly lengthy responses, sensitivity to prompts, and overall subjective evaluation metrics influenced by the evaluator's background. CONCLUSIONS: LLMs exhibit elements of cognitive empathy, recognizing emotions and providing emotionally supportive responses in various contexts. Since social skills are an integral part of intelligence, these advancements bring LLMs closer to human-like interactions and expand their potential use in applications requiring emotional intelligence. However, there remains room for improvement in both the performance of these models and the evaluation strategies used for assessing soft skills.
131	doi:10.1093/jamiaopen/ooae098	2024	Literature search sandbox: a large language model that generates search queries for systematic reviews	Gaelen P. Adam	JAMIA Open	18	L1-openalex-llm-for-lit-search	1	SCREEN	exclude [E6] uses LLMs as a research subject, not as a review tool	Objectives: Development of search queries for systematic reviews (SRs) is time-consuming. In this work, we capitalize on recent advances in large language models (LLMs) and a relatively large dataset of natural language descriptions of reviews and corresponding Boolean searches to generate Boolean search queries from SR titles and key questions. Materials and Methods: We curated a training dataset of 10 346 SR search queries registered in PROSPERO. We used this dataset to fine-tune a set of models to generate search queries based on Mistral-Instruct-7b. We evaluated the models quantitatively using an evaluation dataset of 57 SRs and qualitatively through semi-structured interviews with 8 experienced medical librarians. Results: The model-generated search queries had median sensitivity of 85% (interquartile range [IQR] 40%-100%) and number needed to read of 1206 citations (IQR 205-5810). The interviews suggested that the models lack both the necessary sensitivity and precision to be used without scrutiny but could be useful for topic scoping or as initial queries to be refined. Discussion: Future research should focus on improving the dataset with more high-quality search queries, assessing whether fine-tuning the model on other fields, such as the population and intervention, improves performance, and exploring the addition of interactivity to the interface. Conclusions: The datasets developed for this project can be used to train and evaluate LLMs that map review descriptions to Boolean search queries. The models cannot replace thoughtful search query design but may be useful in providing suggestions for key words and the framework for the query.
132	doi:10.1111/medu.15402	2024	A systematic review of large language models and their implications in medical education	Harrison Lucas	Medical Education	284	L1-openalex-llm-for-lit-search	1	SCREEN	exclude [E6] uses LLMs as a research subject, not as a review tool	INTRODUCTION: In the past year, the use of large language models (LLMs) has generated significant interest and excitement because of their potential to revolutionise various fields, including medical education for aspiring physicians. Although medical students undergo a demanding educational process to become competent health care professionals, the emergence of LLMs presents a promising solution to challenges like information overload, time constraints and pressure on clinical educators. However, integrating LLMs into medical education raises critical concerns and challenges for educators, professionals and students. This systematic review aims to explore LLM applications in medical education, specifically their impact on medical students' learning experiences. METHODS: A systematic search was performed in PubMed, Web of Science and Embase for articles discussing the applications of LLMs in medical education using selected keywords related to LLMs and medical education, from the time of ChatGPT's debut until February 2024. Only articles available in full text or English were reviewed. The credibility of each study was critically appraised by two independent reviewers. RESULTS: The systematic review identified 166 studies, of which 40 were found by review to be relevant to the study. Among the 40 relevant studies, key themes included LLM capabilities, benefits such as personalised learning and challenges regarding content accuracy. Importantly, 42.5% of these studies specifically evaluated LLMs in a novel way, including ChatGPT, in contexts such as medical exams and clinical/biomedical information, highlighting their potential in replicating human-level performance in medical knowledge. The remaining studies broadly discussed the prospective role of LLMs in medical education, reflecting a keen interest in their future potential despite current constraints. CONCLUSIONS: The responsible implementation of LLMs in medical education offers a promising opportunity to enhance learning experiences. However, ensuring information accuracy, emphasising skill-building and maintaining ethical safeguards are crucial. Continuous critical evaluation and interdisciplinary collaboration are essential for the appropriate integration of LLMs in medical education.
133	doi:10.1016/j.csi.2024.103917	2024	Large language models for code completion: A systematic literature review	Rasha Ahmad Husein	Computer Standards & Interfaces	72	L1-openalex-llm-for-lit-search	1	SCREEN	exclude [E6] uses LLMs as a research subject, not as a review tool	Code completion serves as a fundamental aspect of modern software development, improving developers' coding processes. Integrating code completion tools into an Integrated Development Environment (IDE) or code editor enhances the coding process and boosts productivity by reducing errors and speeding up code writing while reducing cognitive load. This is achieved by predicting subsequent tokens, such as keywords, variable names, types, function names, operators, and more. Different techniques can achieve code completion, and recent research has focused on Deep Learning methods, particularly Large Language Models (LLMs) utilizing Transformer algorithms. While several research papers have focused on the use of LLMs for code completion, these studies are fragmented, and there is no systematic overview of the use of LLMs for code completion. Therefore, we aimed to perform a Systematic Literature Review (SLR) study to investigate how LLMs have been applied for code completion so far. We have formulated several research questions to address how LLMs have been integrated for code completion-related tasks and to assess the efficacy of these LLMs in the context of code completion. To achieve this, we retrieved 244 papers from scientific databases using auto-search and specific keywords, finally selecting 23 primary studies based on an SLR methodology for in-depth analysis. This SLR study categorizes the granularity levels of code completion achieved by utilizing LLMs in IDEs, explores the existing issues in current code completion systems, how LLMs address these challenges, and the pre-training and fine-tuning methods employed. Additionally, this study identifies open research problems and outlines future research directions. Our analysis reveals that LLMs significantly enhance code completion performance across several programming languages and contexts, and their capability to predict relevant code snippets based on context and partial input boosts developer productivity substantially.
134	doi:10.2139/ssrn.4943759	2024	Accuracy of Large Language Models for Literature Screening in Systematic Reviews and Meta-Analyses	Zhang-Yi Dai	SSRN Electronic Journal	6	L1-openalex-llm-for-lit-search	1	SCREEN	exclude [E6] uses LLMs as a research subject, not as a review tool
135	doi:10.2196/57400	2024	Large Language Models for Mental Health Applications: Systematic Review	Zhijun Guo	JMIR Mental Health	269	L1-openalex-llm-for-lit-search	1	SCREEN	exclude [E6] uses LLMs as a research subject, not as a review tool	BACKGROUND: Large language models (LLMs) are advanced artificial neural networks trained on extensive datasets to accurately understand and generate natural language. While they have received much attention and demonstrated potential in digital health, their application in mental health, particularly in clinical settings, has generated considerable debate. OBJECTIVE: This systematic review aims to critically assess the use of LLMs in mental health, specifically focusing on their applicability and efficacy in early screening, digital interventions, and clinical settings. By systematically collating and assessing the evidence from current studies, our work analyzes models, methodologies, data sources, and outcomes, thereby highlighting the potential of LLMs in mental health, the challenges they present, and the prospects for their clinical use. METHODS: Adhering to the PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) guidelines, this review searched 5 open-access databases: MEDLINE (accessed by PubMed), IEEE Xplore, Scopus, JMIR, and ACM Digital Library. Keywords used were (mental health OR mental illness OR mental disorder OR psychiatry) AND (large language models). This study included articles published between January 1, 2017, and April 30, 2024, and excluded articles published in languages other than English. RESULTS: In total, 40 articles were evaluated, including 15 (38%) articles on mental health conditions and suicidal ideation detection through text analysis, 7 (18%) on the use of LLMs as mental health conversational agents, and 18 (45%) on other applications and evaluations of LLMs in mental health. LLMs show good effectiveness in detecting mental health issues and providing accessible, destigmatized eHealth services. However, assessments also indicate that the current risks associated with clinical use might surpass their benefits. These risks include inconsistencies in generated text; the production of hallucinations; and the absence of a comprehensive, benchmarked ethical framework. CONCLUSIONS: This systematic review examines the clinical applications of LLMs in mental health, highlighting their potential and inherent risks. The study identifies several issues: the lack of multilingual datasets annotated by experts, concerns regarding the accuracy and reliability of generated content, challenges in interpretability due to the "black box" nature of LLMs, and ongoing ethical dilemmas. These ethical concerns include the absence of a clear, benchmarked ethical framework; data privacy issues; and the potential for overreliance on LLMs by both physicians and patients, which could compromise traditional medical practices. As a result, LLMs should not be considered substitutes for professional mental health services. However, the rapid development of LLMs underscores their potential as valuable clinical aids, emphasizing the need for continued research and development in this area. TRIAL REGISTRATION: PROSPERO CRD42024508617; https://www.crd.york.ac.uk/prospero/display_record.php?RecordID=508617.
136	doi:10.2196/22769	2024	Applications and Concerns of ChatGPT and Other Conversational Large Language Models in Health Care: Systematic Review	Leyao Wang	Journal of Medical Internet Research	172	L1-openalex-llm-for-lit-search	1	SCREEN	exclude [E6] uses LLMs as a research subject, not as a review tool	BACKGROUND: The launch of ChatGPT (OpenAI) in November 2022 attracted public attention and academic interest to large language models (LLMs), facilitating the emergence of many other innovative LLMs. These LLMs have been applied in various fields, including health care. Numerous studies have since been conducted regarding how to use state-of-the-art LLMs in health-related scenarios. OBJECTIVE: This review aims to summarize applications of and concerns regarding conversational LLMs in health care and provide an agenda for future research in this field. METHODS: We used PubMed, ACM, and the IEEE digital libraries as primary sources for this review. We followed the guidance of PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) to screen and select peer-reviewed research articles that (1) were related to health care applications and conversational LLMs and (2) were published before September 1, 2023, the date when we started paper collection. We investigated these papers and classified them according to their applications and concerns. RESULTS: Our search initially identified 820 papers according to targeted keywords, out of which 65 (7.9%) papers met our criteria and were included in the review. The most popular conversational LLM was ChatGPT (60/65, 92% of papers), followed by Bard (Google LLC; 1/65, 2% of papers), LLaMA (Meta; 1/65, 2% of papers), and other LLMs (6/65, 9% papers). These papers were classified into four categories of applications: (1) summarization, (2) medical knowledge inquiry, (3) prediction (eg, diagnosis, treatment recommendation, and drug synergy), and (4) administration (eg, documentation and information collection), and four categories of concerns: (1) reliability (eg, training data quality, accuracy, interpretability, and consistency in responses), (2) bias, (3) privacy, and (4) public acceptability. There were 49 (75%) papers using LLMs for either summarization or medical knowledge inquiry, or both, and there are 58 (89%) papers expressing concerns about either reliability or bias, or both. We found that conversational LLMs exhibited promising results in summarization and providing general medical knowledge to patients with a relatively high accuracy. However, conversational LLMs such as ChatGPT are not always able to provide reliable answers to complex health-related tasks (eg, diagnosis) that require specialized domain expertise. While bias or privacy issues are often noted as concerns, no experiments in our reviewed papers thoughtfully examined how conversational LLMs lead to these issues in health care research. CONCLUSIONS: Future studies should focus on improving the reliability of LLM applications in complex health-related tasks, as well as investigating the mechanisms of how LLM applications bring bias and privacy issues. Considering the vast accessibility of LLMs, legal, social, and technical efforts are all needed to address concerns about LLMs to promote, improve, and regularize the application of LLMs in health care.
137	doi:10.3389/fpsyt.2024.1422807	2024	Applications of large language models in psychiatry: a systematic review	Mahmud Omar	Frontiers in Psychiatry	85	L1-openalex-llm-for-lit-search	1	SCREEN	exclude [E6] uses LLMs as a research subject, not as a review tool	Background: With their unmatched ability to interpret and engage with human language and context, large language models (LLMs) hint at the potential to bridge AI and human cognitive processes. This review explores the current application of LLMs, such as ChatGPT, in the field of psychiatry. Methods: We followed PRISMA guidelines and searched through PubMed, Embase, Web of Science, and Scopus, up until March 2024. Results: From 771 retrieved articles, we included 16 that directly examine LLMs' use in psychiatry. LLMs, particularly ChatGPT and GPT-4, showed diverse applications in clinical reasoning, social media, and education within psychiatry. They can assist in diagnosing mental health issues, managing depression, evaluating suicide risk, and supporting education in the field. However, our review also points out their limitations, such as difficulties with complex cases and potential underestimation of suicide risks. Conclusion: Early research in psychiatry reveals LLMs' versatile applications, from diagnostic support to educational roles. Given the rapid pace of advancement, future investigations are poised to explore the extent to which these models might redefine traditional roles in mental health care.
138	doi:10.1101/2024.04.26.24306390	2024	A Systematic Review of ChatGPT and Other Conversational Large Language Models in Healthcare	Leyao Wang	medRxiv	41	L1-openalex-llm-for-lit-search	1	SCREEN	exclude [E6] uses LLMs as a research subject, not as a review tool	Background: The launch of the Chat Generative Pre-trained Transformer (ChatGPT) in November 2022 has attracted public attention and academic interest to large language models (LLMs), facilitating the emergence of many other innovative LLMs. These LLMs have been applied in various fields, including healthcare. Numerous studies have since been conducted regarding how to employ state-of-the-art LLMs in health-related scenarios to assist patients, doctors, and public health administrators. Objective: This review aims to summarize the applications and concerns of applying conversational LLMs in healthcare and provide an agenda for future research on LLMs in healthcare. Methods: , 2023, the date when we started paper collection and screening. We investigated these papers and classified them according to their applications and concerns. Results: Our search initially identified 820 papers according to targeted keywords, out of which 65 papers met our criteria and were included in the review. The most popular conversational LLM was ChatGPT from OpenAI (60), followed by Bard from Google (1), Large Language Model Meta AI (LLaMA) from Meta (1), and other LLMs (5). These papers were classified into four categories in terms of their applications: 1) summarization, 2) medical knowledge inquiry, 3) prediction, and 4) administration, and four categories of concerns: 1) reliability, 2) bias, 3) privacy, and 4) public acceptability. There are 49 (75%) research papers using LLMs for summarization and/or medical knowledge inquiry, and 58 (89%) research papers expressing concerns about reliability and/or bias. We found that conversational LLMs exhibit promising results in summarization and providing medical knowledge to patients with a relatively high accuracy. However, conversational LLMs like ChatGPT are not able to provide reliable answers to complex health-related tasks that require specialized domain expertise. Additionally, no experiments in our reviewed papers have been conducted to thoughtfully examine how conversational LLMs lead to bias or privacy issues in healthcare research. Conclusions: Future studies should focus on improving the reliability of LLM applications in complex health-related tasks, as well as investigating the mechanisms of how LLM applications brought bias and privacy issues. Considering the vast accessibility of LLMs, legal, social, and technical efforts are all needed to address concerns about LLMs to promote, improve, and regularize the application of LLMs in healthcare.
139	doi:10.1186/s12909-024-05239-y	2024	Large language models for generating medical examinations: systematic review	Yaara Artsi	BMC Medical Education	126	L1-openalex-llm-for-lit-search	1	SCREEN	exclude [E6] uses LLMs as a research subject, not as a review tool	BACKGROUND: Writing multiple choice questions (MCQs) for the purpose of medical exams is challenging. It requires extensive medical knowledge, time and effort from medical educators. This systematic review focuses on the application of large language models (LLMs) in generating medical MCQs. METHODS: The authors searched for studies published up to November 2023. Search terms focused on LLMs generated MCQs for medical examinations. Non-English, out of year range and studies not focusing on AI generated multiple-choice questions were excluded. MEDLINE was used as a search database. Risk of bias was evaluated using a tailored QUADAS-2 tool. RESULTS: Overall, eight studies published between April 2023 and October 2023 were included. Six studies used Chat-GPT 3.5, while two employed GPT 4. Five studies showed that LLMs can produce competent questions valid for medical exams. Three studies used LLMs to write medical questions but did not evaluate the validity of the questions. One study conducted a comparative analysis of different models. One other study compared LLM-generated questions with those written by humans. All studies presented faulty questions that were deemed inappropriate for medical exams. Some questions required additional modifications in order to qualify. CONCLUSIONS: LLMs can be used to write MCQs for medical examinations. However, their limitations cannot be ignored. Further study in this field is essential and more conclusive evidence is needed. Until then, LLMs may serve as a supplementary tool for writing medical examinations. 2 studies were at high risk of bias. The study followed the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) guidelines.
140	doi:10.1001/jamanetworkopen.2024.20496	2024	Performance of a Large Language Model in Screening Citations	Takehiko Oami	JAMA Network Open	85	L1-openalex-llm-for-lit-search	1	SCREEN	maybe [LLM] LLM citation screening performance	Importance: Large language models (LLMs) are promising as tools for citation screening in systematic reviews. However, their applicability has not yet been determined. Objective: To evaluate the accuracy and efficiency of an LLM in title and abstract literature screening. Design, Setting, and Participants: This prospective diagnostic study used the data from the title and abstract screening process for 5 clinical questions (CQs) in the development of the Japanese Clinical Practice Guidelines for Management of Sepsis and Septic Shock. The LLM decided to include or exclude citations based on the inclusion and exclusion criteria in terms of patient, population, problem; intervention; comparison; and study design of the selected CQ and was compared with the conventional method for title and abstract screening. This study was conducted from January 7 to 15, 2024. Exposures: LLM (GPT-4 Turbo)-assisted citation screening or the conventional method. Main Outcomes and Measures: The sensitivity and specificity of the LLM-assisted screening process was calculated, and the full-text screening result using the conventional method was set as the reference standard in the primary analysis. Pooled sensitivity and specificity were also estimated, and screening times of the 2 methods were compared. Results: In the conventional citation screening process, 8 of 5634 publications in CQ 1, 4 of 3418 in CQ 2, 4 of 1038 in CQ 3, 17 of 4326 in CQ 4, and 8 of 2253 in CQ 5 were selected. In the primary analysis of 5 CQs, LLM-assisted citation screening demonstrated an integrated sensitivity of 0.75 (95% CI, 0.43 to 0.92) and specificity of 0.99 (95% CI, 0.99 to 0.99). Post hoc modifications to the command prompt improved the integrated sensitivity to 0.91 (95% CI, 0.77 to 0.97) without substantially compromising specificity (0.98 [95% CI, 0.96 to 0.99]). Additionally, LLM-assisted screening was associated with reduced time for processing 100 studies (1.3 minutes vs 17.2 minutes for conventional screening methods; mean difference, -15.25 minutes [95% CI, -17.70 to -12.79 minutes]). Conclusions and Relevance: In this prospective diagnostic study investigating the performance of LLM-assisted citation screening, the model demonstrated acceptable sensitivity and reasonably high specificity with reduced processing time. This novel method could potentially enhance efficiency and reduce workload in systematic reviews.
141	doi:10.3390/jcm13113041	2024	Clinical and Surgical Applications of Large Language Models: A Systematic Review	Sophia M. Pressman	Journal of Clinical Medicine	106	L1-openalex-llm-for-lit-search	1	SCREEN	exclude [E6] uses LLMs as a research subject, not as a review tool	Background: Large language models (LLMs) represent a recent advancement in artificial intelligence with medical applications across various healthcare domains. The objective of this review is to highlight how LLMs can be utilized by clinicians and surgeons in their everyday practice. Methods: A systematic review was conducted following the Preferred Reporting Items for Systematic Reviews and Meta-Analyses guidelines. Six databases were searched to identify relevant articles. Eligibility criteria emphasized articles focused primarily on clinical and surgical applications of LLMs. Results: The literature search yielded 333 results, with 34 meeting eligibility criteria. All articles were from 2023. There were 14 original research articles, four letters, one interview, and 15 review articles. These articles covered a wide variety of medical specialties, including various surgical subspecialties. Conclusions: LLMs have the potential to enhance healthcare delivery. In clinical settings, LLMs can assist in diagnosis, treatment guidance, patient triage, physician knowledge augmentation, and administrative tasks. In surgical settings, LLMs can assist surgeons with documentation, surgical planning, and intraoperative guidance. However, addressing their limitations and concerns, particularly those related to accuracy and biases, is crucial. LLMs should be viewed as tools to complement, not replace, the expertise of healthcare professionals.
142	doi:10.2196/66648	2024	Large Language Models in Gastroenterology: Systematic Review	Eun Jeong Gong	Journal of Medical Internet Research	37	L1-openalex-llm-for-lit-search	1	SCREEN	exclude [E6] uses LLMs as a research subject, not as a review tool	BACKGROUND: As health care continues to evolve with technological advancements, the integration of artificial intelligence into clinical practices has shown promising potential to enhance patient care and operational efficiency. Among the forefront of these innovations are large language models (LLMs), a subset of artificial intelligence designed to understand, generate, and interact with human language at an unprecedented scale. OBJECTIVE: This systematic review describes the role of LLMs in improving diagnostic accuracy, automating documentation, and advancing specialist education and patient engagement within the field of gastroenterology and gastrointestinal endoscopy. METHODS: Core databases including MEDLINE through PubMed, Embase, and Cochrane Central registry were searched using keywords related to LLMs (from inception to April 2024). Studies were included if they satisfied the following criteria: (1) any type of studies that investigated the potential role of LLMs in the field of gastrointestinal endoscopy or gastroenterology, (2) studies published in English, and (3) studies in full-text format. The exclusion criteria were as follows: (1) studies that did not report the potential role of LLMs in the field of gastrointestinal endoscopy or gastroenterology, (2) case reports and review papers, (3) ineligible research objects (eg, animals or basic research), and (4) insufficient data regarding the potential role of LLMs. Risk of Bias in Non-Randomized Studies-of Interventions was used to evaluate the quality of the identified studies. RESULTS: Overall, 21 studies on the potential role of LLMs in gastrointestinal disorders were included in the systematic review, and narrative synthesis was done because of heterogeneity in the specified aims and methodology in each included study. The overall risk of bias was low in 5 studies and moderate in 16 studies. The ability of LLMs to spread general medical information, offer advice for consultations, generate procedure reports automatically, or draw conclusions about the presumptive diagnosis of complex medical illnesses was demonstrated by the systematic review. Despite promising benefits, such as increased efficiency and improved patient outcomes, challenges related to data privacy, accuracy, and interdisciplinary collaboration remain. CONCLUSIONS: We highlight the importance of navigating these challenges to fully leverage LLMs in transforming gastrointestinal endoscopy practices. TRIAL REGISTRATION: PROSPERO 581772; https://www.crd.york.ac.uk/prospero/.
143	doi:10.2196/59607	2024	Examining the Role of Large Language Models in Orthopedics: Systematic Review	Cheng Zhang	Journal of Medical Internet Research	64	L1-openalex-llm-for-lit-search	1	SCREEN	exclude [E6] uses LLMs as a research subject, not as a review tool	BACKGROUND: Large language models (LLMs) can understand natural language and generate corresponding text, images, and even videos based on prompts, which holds great potential in medical scenarios. Orthopedics is a significant branch of medicine, and orthopedic diseases contribute to a significant socioeconomic burden, which could be alleviated by the application of LLMs. Several pioneers in orthopedics have conducted research on LLMs across various subspecialties to explore their performance in addressing different issues. However, there are currently few reviews and summaries of these studies, and a systematic summary of existing research is absent. OBJECTIVE: The objective of this review was to comprehensively summarize research findings on the application of LLMs in the field of orthopedics and explore the potential opportunities and challenges. METHODS: PubMed, Embase, and Cochrane Library databases were searched from January 1, 2014, to February 22, 2024, with the language limited to English. The terms, which included variants of "large language model," "generative artificial intelligence," "ChatGPT," and "orthopaedics," were divided into 2 categories: large language model and orthopedics. After completing the search, the study selection process was conducted according to the inclusion and exclusion criteria. The quality of the included studies was assessed using the revised Cochrane risk-of-bias tool for randomized trials and CONSORT-AI (Consolidated Standards of Reporting Trials-Artificial Intelligence) guidance. Data extraction and synthesis were conducted after the quality assessment. RESULTS: A total of 68 studies were selected. The application of LLMs in orthopedics involved the fields of clinical practice, education, research, and management. Of these 68 studies, 47 (69%) focused on clinical practice, 12 (18%) addressed orthopedic education, 8 (12%) were related to scientific research, and 1 (1%) pertained to the field of management. Of the 68 studies, only 8 (12%) recruited patients, and only 1 (1%) was a high-quality randomized controlled trial. ChatGPT was the most commonly mentioned LLM tool. There was considerable heterogeneity in the definition, measurement, and evaluation of the LLMs' performance across the different studies. For diagnostic tasks alone, the accuracy ranged from 55% to 93%. When performing disease classification tasks, ChatGPT with GPT-4's accuracy ranged from 2% to 100%. With regard to answering questions in orthopedic examinations, the scores ranged from 45% to 73.6% due to differences in models and test selections. CONCLUSIONS: LLMs cannot replace orthopedic professionals in the short term. However, using LLMs as copilots could be a potential approach to effectively enhance work efficiency at present. More high-quality clinical trials are needed in the future, aiming to identify optimal applications of LLMs and advance orthopedics toward higher efficiency and precision.
144	doi:10.1101/2024.04.15.24305869	2024	A Systematic Review of Testing and Evaluation of Healthcare Applications of Large Language Models (LLMs)	Suhana Bedi	medRxiv	48	L1-openalex-llm-for-lit-search	1	SCREEN	exclude [E6] uses LLMs as a research subject, not as a review tool	1 Abstract Importance Large Language Models (LLMs) can assist in a wide range of healthcare-related activities. Current approaches to evaluating LLMs make it difficult to identify the most impactful LLM application areas. Objective To summarize the current evaluation of LLMs in healthcare in terms of 5 components: evaluation data type, healthcare task, Natural Language Processing (NLP)/Natural Language Understanding (NLU) task, dimension of evaluation, and medical specialty. Data Sources A systematic search of PubMed and Web of Science was performed for studies published between 01-01-2022 and 02-19-2024. Study Selection Studies evaluating one or more LLMs in healthcare. Data Extraction and Synthesis Three independent reviewers categorized 519 studies in terms of data used in the evaluation, the healthcare tasks (the what) and the NLP/NLU tasks (the how) examined, the dimension(s) of evaluation, and the medical specialty studied. Results Only 5% of reviewed studies utilized real patient care data for LLM evaluation. The most popular healthcare tasks were assessing medical knowledge (e.g. answering medical licensing exam questions, 44.5%), followed by making diagnoses (19.5%), and educating patients (17.7%). Administrative tasks such as assigning provider billing codes (0.2%), writing prescriptions (0.2%), generating clinical referrals (0.6%) and clinical notetaking (0.8%) were less studied. For NLP/NLU tasks, the vast majority of studies examined question answering (84.2%). Other tasks such as summarization (8.9%), conversational dialogue (3.3%), and translation (3.1%) were infrequent. Almost all studies (95.4%) used accuracy as the primary dimension of evaluation; fairness, bias and toxicity (15.8%), robustness (14.8%), deployment considerations (4.6%), and calibration and uncertainty (1.2%) were infrequently measured. Finally, in terms of medical specialty area, most studies were in internal medicine (42%), surgery (11.4%) and ophthalmology (6.9%), with nuclear medicine (0.6%), physical medicine (0.4%) and medical genetics (0.2%) being the least represented. Conclusions and Relevance Existing evaluations of LLMs mostly focused on accuracy of question answering for medical exams, without consideration of real patient care data. Dimensions like fairness, bias and toxicity, robustness, and deployment considerations received limited attention. To draw meaningful conclusions and improve LLM adoption, future studies need to establish a standardized set of LLM applications and evaluation dimensions, perform evaluations using data from routine care, and broaden testing to include administrative tasks as well as multiple medical specialties. Key Points Question How are healthcare applications of large language models (LLMs) currently evaluated? Findings Studies rarely used real patient care data for LLM evaluation. Administrative tasks such as generating provider billing codes and writing prescriptions were understudied. Natural Language Processing (NLP)/Natural Language Understanding (NLU) tasks like summarization, conversational dialogue, and translation were infrequently explored. Accuracy was the predominant dimension of evaluation, while fairness, bias and toxicity assessments were neglected. Evaluations in specialized fields, such as nuclear medicine and medical genetics were rare. Meaning Current LLM assessments in healthcare remain shallow and fragmented. To draw concrete insights on their performance, evaluations need to use real patient care data across a broad range of healthcare and NLP/NLU tasks and medical specialties with standardized dimensions of evaluation.
145	doi:10.1002/jrsm.1715	2024	Can large language models replace humans in systematic reviews? Evaluating GPT ‐4's efficacy in screening and extracting data from peer‐reviewed and grey literature in multiple languages	Qusai Khraisha	Research Synthesis Methods	266	L1-openalex-llm-for-lit-search	1	SCREEN	include [LLM] GPT-4 vs humans across SR stages; pre-registered; caution	Systematic reviews are vital for guiding practice, research and policy, although they are often slow and labour-intensive. Large language models (LLMs) could speed up and automate systematic reviews, but their performance in such tasks has yet to be comprehensively evaluated against humans, and no study has tested Generative Pre-Trained Transformer (GPT)-4, the biggest LLM so far. This pre-registered study uses a "human-out-of-the-loop" approach to evaluate GPT-4's capability in title/abstract screening, full-text review and data extraction across various literature types and languages. Although GPT-4 had accuracy on par with human performance in some tasks, results were skewed by chance agreement and dataset imbalance. Adjusting for these caused performance scores to drop across all stages: for data extraction, performance was moderate, and for screening, it ranged from none in highly balanced literature datasets (~1:1) to moderate in those datasets where the ratio of inclusion to exclusion in studies was imbalanced (~1:3). When screening full-text literature using highly reliable prompts, GPT-4's performance was more robust, reaching "human-like" levels. Although our findings indicate that, currently, substantial caution should be exercised if LLMs are being used to conduct systematic reviews, they also offer preliminary evidence that, for certain review tasks delivered under specific conditions, LLMs can rival human performance.
146	doi:10.3389/fdsfr.2024.1379260	2024	Assessing the performance of large language models in literature screening for pharmacovigilance: a comparative study	Dan Li	Frontiers in Drug Safety and Regulation	18	L1-openalex-llm-for-lit-search	1	SCREEN	exclude [E6] uses LLMs as a research subject, not as a review tool	Pharmacovigilance plays a crucial role in ensuring the safety of pharmaceutical products. It involves the systematic monitoring of adverse events and the detection of potential safety concerns related to drugs. Manual literature screening for pharmacovigilance related articles is a labor-intensive and time-consuming task, requiring streamlined solutions to cope with the continuous growth of literature. The primary objective of this study is to assess the performance of Large Language Models (LLMs) in automating literature screening for pharmacovigilance, aiming to enhance the process by identifying relevant articles more effectively. This study represents a novel application of LLMs including OpenAI's GPT-3.5, GPT-4, and Anthropic's Claude2, in the field of pharmacovigilance, evaluating their ability to categorize medical publications as relevant or irrelevant for safety signal reviews. Our analysis encompassed N-shot learning, chain-of-thought reasoning, and evaluating metrics, with a focus on factors impacting accuracy. The findings highlight the promising potential of LLMs in literature screening, achieving a reproducibility of 93%, sensitivity of 97%, and specificity of 67% showcasing notable strengths in terms of reproducibility and sensitivity, although with moderate specificity. Notably, performance improved when models were provided examples consisting of abstracts, labels, and corresponding reasoning explanations. Moreover, our exploration identified several potential contributing factors influencing prediction outcomes. These factors encompassed the choice of key words and prompts, the balance of the examples, and variations in reasoning explanations. By configuring advanced LLMs for efficient screening of extensive literature databases, this study underscores the transformative potential of these models in drug safety monitoring. Furthermore, these insights gained from this study can inform the development of automated systems for pharmacovigilance, contributing to the ongoing efforts to ensure the safety and efficacy of pharmacovigilance products.
147	doi:10.1016/j.isci.2024.109713	2024	The application of large language models in medicine: A scoping review	Xiangbin Meng	iScience	414	L1-openalex-llm-for-lit-search	1	SCREEN	exclude [E6] uses LLMs as a research subject, not as a review tool	This study systematically reviewed the application of large language models (LLMs) in medicine, analyzing 550 selected studies from a vast literature search. LLMs like ChatGPT transformed healthcare by enhancing diagnostics, medical writing, education, and project management. They assisted in drafting medical documents, creating training simulations, and streamlining research processes. Despite their growing utility in assisted diagnosis and improving doctor-patient communication, challenges persisted, including limitations in contextual understanding and the risk of over-reliance. The surge in LLM-related research indicated a focus on medical writing, diagnostics, and patient communication, but highlighted the need for careful integration, considering validation, ethical concerns, and the balance with traditional medical practice. Future research directions suggested a focus on multimodal LLMs, deeper algorithmic understanding, and ensuring responsible, effective use in healthcare.
148	doi:10.2196/56780	2024	Potential Roles of Large Language Models in the Production of Systematic Reviews and Meta-Analyses	Xufei Luo	Journal of Medical Internet Research	76	L1-openalex-llm-for-lit-search	1	SCREEN	exclude [E6] uses LLMs as a research subject, not as a review tool	Large language models (LLMs) such as ChatGPT have become widely applied in the field of medical research. In the process of conducting systematic reviews, similar tools can be used to expedite various steps, including defining clinical questions, performing the literature search, document screening, information extraction, and language refinement, thereby conserving resources and enhancing efficiency. However, when using LLMs, attention should be paid to transparent reporting, distinguishing between genuine and false content, and avoiding academic misconduct. In this viewpoint, we highlight the potential roles of LLMs in the creation of systematic reviews and meta-analyses, elucidating their advantages, limitations, and future research directions, aiming to provide insights and guidance for authors planning systematic reviews and meta-analyses.
149	doi:10.1186/s13643-024-02609-x	2024	Evaluating the effectiveness of large language models in abstract screening: a comparative analysis	Michael Li	Systematic Reviews	127	L1-openalex-llm-for-lit-search	1	SCREEN	exclude [E6] uses LLMs as a research subject, not as a review tool	OBJECTIVE: This study aimed to evaluate the performance of large language models (LLMs) in the task of abstract screening in systematic review and meta-analysis studies, exploring their effectiveness, efficiency, and potential integration into existing human expert-based workflows. METHODS: We developed automation scripts in Python to interact with the APIs of several LLM tools, including ChatGPT v4.0, ChatGPT v3.5, Google PaLM 2, and Meta Llama 2, and latest tools including ChatGPT v4.0 turbo, ChatGPT v3.5 turbo, Google Gemini 1.0 pro, Meta Llama 3, and Claude 3. This study focused on three databases of abstracts and used them as benchmarks to evaluate the performance of these LLM tools in terms of sensitivity, specificity, and overall accuracy. The results of the LLM tools were compared to human-curated inclusion decisions, gold standard for systematic review and meta-analysis studies. RESULTS: Different LLM tools had varying abilities in abstract screening. Chat GPT v4.0 demonstrated remarkable performance, with balanced sensitivity and specificity, and overall accuracy consistently reaching or exceeding 90%, indicating a high potential for LLMs in abstract screening tasks. The study found that LLMs could provide reliable results with minimal human effort and thus serve as a cost-effective and efficient alternative to traditional abstract screening methods. CONCLUSION: While LLM tools are not yet ready to completely replace human experts in abstract screening, they show great promise in revolutionizing the process. They can serve as autonomous AI reviewers, contribute to collaborative workflows with human experts, and integrate with hybrid approaches to develop custom tools for increased efficiency. As technology continues to advance, LLMs are poised to play an increasingly important role in abstract screening, reshaping the workflow of systematic review and meta-analysis studies.
150	doi:10.1007/s00432-024-05678-6	2024	Utilizing large language models in breast cancer management: systematic review	Vera Sorin	Journal of Cancer Research and Clinical Oncology	45	L1-openalex-llm-for-lit-search	1	SCREEN	exclude [E6] uses LLMs as a research subject, not as a review tool	PURPOSE: Despite advanced technologies in breast cancer management, challenges remain in efficiently interpreting vast clinical data for patient-specific insights. We reviewed the literature on how large language models (LLMs) such as ChatGPT might offer solutions in this field. METHODS: We searched MEDLINE for relevant studies published before December 22, 2023. Keywords included: "large language models", "LLM", "GPT", "ChatGPT", "OpenAI", and "breast". The risk bias was evaluated using the QUADAS-2 tool. RESULTS: Six studies evaluating either ChatGPT-3.5 or GPT-4, met our inclusion criteria. They explored clinical notes analysis, guideline-based question-answering, and patient management recommendations. Accuracy varied between studies, ranging from 50 to 98%. Higher accuracy was seen in structured tasks like information retrieval. Half of the studies used real patient data, adding practical clinical value. Challenges included inconsistent accuracy, dependency on the way questions are posed (prompt-dependency), and in some cases, missing critical clinical information. CONCLUSION: LLMs hold potential in breast cancer care, especially in textual information extraction and guideline-driven clinical question-answering. Yet, their inconsistent accuracy underscores the need for careful validation of these models, and the importance of ongoing supervision.
151	doi:10.1016/j.jval.2024.03.1461	2024	MSR28 Use of Large Language Model (LLM) for Full-Text Screening in Systematic Literature Reviews: A Comparative Analysis	H Rathi	Value in Health	2	L1-openalex-llm-for-lit-search	1	SCREEN	exclude [E6] uses LLMs as a research subject, not as a review tool
152	doi:10.1007/978-3-031-56027-9_25	2024	Zero-Shot Generative Large Language Models for Systematic Review Screening Automation	Shuai Wang	Lecture notes in computer science	23	L1-openalex-llm-for-lit-search	1	SCREEN	exclude [E6] uses LLMs as a research subject, not as a review tool	Systematic reviews are crucial for evidence-based medicine as they comprehensively analyse published research findings on specific questions. Conducting such reviews is often resource- and time-intensive, especially in the screening phase, where abstracts of publications are assessed for inclusion in a review. This study investigates the effectiveness of using zero-shot large language models~(LLMs) for automatic screening. We evaluate the effectiveness of eight different LLMs and investigate a calibration technique that uses a predefined recall threshold to determine whether a publication should be included in a systematic review. Our comprehensive evaluation using five standard test collections shows that instruction fine-tuning plays an important role in screening, that calibration renders LLMs practical for achieving a targeted recall, and that combining both with an ensemble of zero-shot models saves significant screening time compared to state-of-the-art approaches.
153	doi:10.1101/2024.09.01.24312887	2024	Large Language Models in Cardiology: A Systematic Review	Moran Gendler	medRxiv	5	L1-openalex-llm-for-lit-search	1	SCREEN	exclude [E6] uses LLMs as a research subject, not as a review tool	Background: Large language models (LLMs) are increasingly used in health care, but their role in cardiology has not yet been systematically evaluated. Objective: This review aimed to assess the applications, performance, and limitations of LLMs across diverse cardiology tasks, including chronic and progressive conditions, acute events, education, and diagnostic testing. Methods: A systematic search was conducted in PubMed and Scopus for studies published up to April 14, 2024, using keywords related to LLMs and cardiology. Studies evaluating LLM outputs in cardiology-related tasks were included. Data were extracted across 5 predefined domains and the risk of bias was assessed using an adapted QUADAS-2 tool (developed by Whiting et al at the University of Bristol). The review protocol was registered in PROSPERO (CRD42024556397). Results: A total of 33 studies contributed quantitative outcome data to a descriptive synthesis. Across chronic conditions, ChatGPT-3.5 (OpenAI) answered 91% (43/47) heart failure questions accurately, although readability often required college-level comprehension. In acute scenarios, Bing Chat omitted key myocardial infarction first aid steps in 25% (5/20) to 45% (9/20) of cases, while cardiac arrest information was rated highly (mean 4.3/5, SD 0.7) but written above recommended reading levels. In physician education tasks, ChatGPT-4 (OpenAI) demonstrated higher accuracy than ChatGPT-3.5, improving from 38% (33/88) to 66% (58/88). In patient education studies, ChatGPT-4 provided scientifically adequate explanations (5.0-6.0/7) comparable to hospital materials but at higher reading levels (11th vs 7th grade). In diagnostic testing, ChatGPT-4 interpreted 91% (36/40) electrocardiogram vignettes correctly, significantly better than emergency physicians (31/40, 77%; P< .001), but showed lower performance in echocardiography. Conclusions: LLMs show meaningful potential in cardiology, especially for education and electrocardiogram interpretation, but performance varies across clinical tasks. Limitations in emergency guidance and readability, as well as small in silico study designs, highlight the need for multimodal models and prospective validation.
154	doi:10.1007/s00701-024-06372-9	2024	Large language models in neurosurgery: a systematic review and meta-analysis	Advait Patil	Acta Neurochirurgica	26	L1-openalex-llm-for-lit-search	1	SCREEN	exclude [E6] uses LLMs as a research subject, not as a review tool
155	doi:10.1111/apt.18058	2024	Systematic review: The use of large language models as medical chatbots in digestive diseases	Mauro Giuffrè	Alimentary Pharmacology & Therapeutics	55	L1-openalex-llm-for-lit-search	1	SCREEN	exclude [E6] uses LLMs as a research subject, not as a review tool	BACKGROUND: Interest in large language models (LLMs), such as OpenAI's ChatGPT, across multiple specialties has grown as a source of patient-facing medical advice and provider-facing clinical decision support. The accuracy of LLM responses for gastroenterology and hepatology-related questions is unknown. AIMS: To evaluate the accuracy and potential safety implications for LLMs for the diagnosis, management and treatment of questions related to gastroenterology and hepatology. METHODS: We conducted a systematic literature search including Cochrane Library, Google Scholar, Ovid Embase, Ovid MEDLINE, PubMed, Scopus and the Web of Science Core Collection to identify relevant articles published from inception until January 28, 2024, using a combination of keywords and controlled vocabulary for LLMs and gastroenterology or hepatology. Accuracy was defined as the percentage of entirely correct answers. RESULTS: Among the 1671 reports screened, we identified 33 full-text articles on using LLMs in gastroenterology and hepatology and included 18 in the final analysis. The accuracy of question-responding varied across different model versions. For example, accuracy ranged from 6.4% to 45.5% with ChatGPT-3.5 and was between 40% and 91.4% with ChatGPT-4. In addition, the absence of standardised methodology and reporting metrics for studies involving LLMs places all the studies at a high risk of bias and does not allow for the generalisation of single-study results. CONCLUSIONS: Current general-purpose LLMs have unacceptably low accuracy on clinical gastroenterology and hepatology tasks, which may lead to adverse patient safety events through incorrect information or triage recommendations, which might overburden healthcare systems or delay necessary care.
156	doi:10.1016/j.jad.2024.11.052	2024	Exploring the efficacy and potential of large language models for depression: A systematic review	Mahmud Omar	Journal of Affective Disorders	52	L1-openalex-llm-for-lit-search	1	SCREEN	exclude [E6] uses LLMs as a research subject, not as a review tool
157	doi:10.1016/j.jval.2024.03.1477	2024	MSR44 How Accurate Are Large Language Models for Abstract Screening in Systematic Literature Reviews?	A Sertel	Value in Health	1	L1-openalex-llm-for-lit-search	1	SCREEN	exclude [E6] uses LLMs as a research subject, not as a review tool
158	doi:10.1016/j.identj.2024.10.014	2024	Large Language Models in Dental Licensing Examinations: Systematic Review and Meta-Analysis	Mingxin Liu	International Dental Journal	41	L1-openalex-llm-for-lit-search	1	SCREEN	exclude [E6] uses LLMs as a research subject, not as a review tool	INTRODUCTION AND AIMS: This study systematically reviews and conducts a meta-analysis to evaluate the performance of various large language models (LLMs) in dental licensing examinations worldwide. The aim is to assess the accuracy of these models in different linguistic and geographical contexts. This will inform their potential application in dental education and diagnostics. METHODS: Following Preferred Reporting Items for Systematic Reviews and Meta-Analyses guidelines, we conducted a comprehensive search across PubMed, Web of Science, and Scopus for studies published from 1 January 2022 to 1 May 2024. Two authors independently reviewed the literature based on the inclusion and exclusion criteria, extracted data, and evaluated the quality of the studies in accordance with the Quality Assessment of Diagnostic Accuracy Studies-2. We conducted qualitative and quantitative analyses to evaluate the performance of LLMs. RESULTS: Eleven studies met the inclusion criteria, encompassing dental licensing examinations from eight countries. GPT-3.5, GPT-4, and Bard achieved integrated accuracy rates of 54%, 72%, and 56%, respectively. GPT-4 outperformed GPT-3.5 and Bard, passing more than half of the dental licensing examinations. Subgroup analyses and meta-regression showed that GPT-3.5 performed significantly better in English-speaking countries. GPT-4's performance, however, remained consistent across different regions. CONCLUSION: LLMs, particularly GPT-4, show potential in dental education and diagnostics, yet their accuracy remains below the threshold required for clinical application. The lack of sufficient training data in dentistry has affected LLMs' accuracy. The reliance on image-based diagnostics also presents challenges. As a result, their accuracy in dental exams is lower compared to medical licensing exams. Additionally, LLMs even provide more detailed explanation for incorrect answer than correct one. Overall, the current LLMs are not yet suitable for use in dental education and clinical diagnosis.
159	doi:10.1101/2024.01.05.24300884	2024	Large language models in simplifying radiological reports: systematic review	Yaara Artsi	medRxiv	9	L1-openalex-llm-for-lit-search	1	SCREEN	exclude [E6] uses LLMs as a research subject, not as a review tool	Abstract Objectives Simplifying medical information to make it understandable for patients, specifically in the case of radiology reports, is challenging. It requires time and effort from medical personnel. This systematic review focuses on the application of large language models (LLMs) in generating simplified radiological imaging reports, as well as answering patient inquiries regarding radiological procedures. Materials and Methods The authors searched for studies published up to January 2024. Search terms focused on LLMs generated simplified radiological reports and answers to patient inquiries regarding radiological procedures. MEDLINE was used as a search database. Results Overall, eight studies published between May 2023 and November 2023 were included. All studies showed that LLMs can produce simplified medical information for patients. Four studies (50%) used GPT-3.5, Two studies (25%) conducted a comparative analysis between GPT-3.5 and GPT-4. One study (12.5%) examined Microsoft Bing. One study (12.5%) utilized GPT-4. Four studies (50%) used LLMs to simplify radiological reports. Four studies (50%) used LLMs to answer patient questions regarding radiological procedures. Only two studies (25%) used patients to evaluate the LLMs output. One study (12.5%) compared their initial prompt with optimized prompt. Five studies (62.5%) showed missing, inaccurate and potentially harmful AI outputs. Conclusion LLMs can be used to simplify medical imaging reports and procedures, for improved patient comprehension. However, their limitations cannot be ignored. Further study in this field is essential and more conclusive evidence is needed.
160	doi:10.1101/2024.06.03.24308405	2024	Evaluating the Efficacy of Large Language Models for Systematic Review and Meta-Analysis Screening	Ronald Luo	medRxiv	8	L1-openalex-llm-for-lit-search	1	SCREEN	exclude [E6] uses LLMs as a research subject, not as a review tool	ABSTRACT Background Systematic reviews and meta-analyses are essential for informed research and policymaking, yet they are typically resource-intensive and time-consuming. Recent advances in artificial intelligence and machine learning offer promising opportunities to streamline these processes. Objective To enhance the efficiency of systematic reviews, we explored the automation of various stages using GPT-3.5 Turbo. We assessed the model’s efficacy and performance by comparing it against three expert-conducted reviews across a comprehensive dataset of 24,534 studies. Methods The model’s performance was evaluated through a comparison with three expert reviews, utilizing a pseudo-K-folds permutation and a one-tailed ANOVA with an alpha level of 0.05 to ensure statistical validity. Key performance metrics such as accuracy, sensitivity, specificity, predictive values, F1-score, and the Matthews correlation coefficient were analyzed using two sets of prompts. Results Our approach significantly streamlined the systematic review process, which typically takes a year, reducing it to a few hours without sacrificing quality. In the initial screening phase, accuracy, specificity, and negative predictive values ranged between 80% and 95%. Sensitivity improved markedly during the second screening phase, demonstrating the model’s robustness when provided with more extensive data. Conclusion While ongoing refinements are needed, this tool represents a significant advancement in research methodologies, potentially making systematic reviews more accessible to a wider range of researchers. Impact Statement Our manuscript presents a novel review screening protocol built using open-source frameworks, which significantly enhances the systematic review process in terms of efficiency and cost-effectiveness. Leveraging the capabilities of GPT and embedding models, our protocol demonstrates the potential to transform a traditionally time-consuming and expensive task into an accelerated and economical operation, all while maintaining high standards of accuracy and reliability. Key Points GPT screening can streamline systematic reviews from a year-long, expensive process to just hours at minimal cost. Validated across different topics, the protocol exhibits high reliability and consistency in study inclusion. The AI-driven process reduces human bias, with prompt optimization considerably improving sensitivity.
161	doi:10.3390/jcm13237480	2024	Exploring the Role of Large Language Models in Melanoma: A Systematic Review	Mor Zarfati	Journal of Clinical Medicine	9	L1-openalex-llm-for-lit-search	1	SCREEN	exclude [E6] uses LLMs as a research subject, not as a review tool	Objective: This systematic review evaluates the current applications, advantages, and challenges of large language models (LLMs) in melanoma care. Methods: A systematic search was conducted in PubMed and Scopus databases for studies published up to 23 July 2024, focusing on the application of LLMs in melanoma. The review adhered to PRISMA guidelines, and the risk of bias was assessed using the modified QUADAS-2 tool. Results: Nine studies were included, categorized into subgroups: patient education, diagnosis, and clinical management. In patient education, LLMs demonstrated high accuracy, though readability often exceeded recommended levels. For diagnosis, multimodal LLMs like GPT-4V showed capabilities in distinguishing melanoma from benign lesions, but accuracy varied, influenced by factors such as image quality and integration of clinical context. Regarding management advice, ChatGPT provided more reliable recommendations compared to other LLMs, but all models lacked depth for individualized decision-making. Conclusions: LLMs, particularly multimodal models, show potential in improving melanoma care. However, current applications require further refinement and validation. Future studies should explore fine-tuning these models on large, diverse dermatological databases and incorporate expert knowledge to address limitations such as generalizability across different populations and skin types.
162	doi:10.3390/healthcare12080825	2024	AI and Ethics: A Systematic Review of the Ethical Considerations of Large Language Model Use in Surgery Research	Sophia M. Pressman	Healthcare	111	L1-openalex-llm-for-lit-search	1	SCREEN	exclude [E6] uses LLMs as a research subject, not as a review tool	INTRODUCTION: As large language models receive greater attention in medical research, the investigation of ethical considerations is warranted. This review aims to explore surgery literature to identify ethical concerns surrounding these artificial intelligence models and evaluate how autonomy, beneficence, nonmaleficence, and justice are represented within these ethical discussions to provide insights in order to guide further research and practice. METHODS: A systematic review was conducted in accordance with the Preferred Reporting Items for Systematic Reviews and Meta-Analyses guidelines. Five electronic databases were searched in October 2023. Eligible studies included surgery-related articles that focused on large language models and contained adequate ethical discussion. Study details, including specialty and ethical concerns, were collected. RESULTS: The literature search yielded 1179 articles, with 53 meeting the inclusion criteria. Plastic surgery, orthopedic surgery, and neurosurgery were the most represented surgical specialties. Autonomy was the most explicitly cited ethical principle. The most frequently discussed ethical concern was accuracy (n = 45, 84.9%), followed by bias, patient confidentiality, and responsibility. CONCLUSION: The ethical implications of using large language models in surgery are complex and evolving. The integration of these models into surgery necessitates continuous ethical discourse to ensure responsible and ethical use, balancing technological advancement with human dignity and safety.
163	doi:10.2196/preprints.57400	2024	Large Language Models for Mental Health Applications: Systematic Review (Preprint)	Zhijun Guo		24	L1-openalex-llm-for-lit-search	1	SCREEN	exclude [E6] uses LLMs as a research subject, not as a review tool	BACKGROUND Large language models (LLMs) are advanced artificial neural networks trained on extensive datasets to accurately understand and generate natural language. While they have received much attention and demonstrated potential in digital health, their application in mental health, particularly in clinical settings, has generated considerable debate. OBJECTIVE This systematic review aims to critically assess the use of LLMs in mental health, specifically focusing on their applicability and efficacy in early screening, digital interventions, and clinical settings. By systematically collating and assessing the evidence from current studies, our work analyzes models, methodologies, data sources, and outcomes, thereby highlighting the potential of LLMs in mental health, the challenges they present, and the prospects for their clinical use. METHODS Adhering to the PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) guidelines, this review searched 5 open-access databases: MEDLINE (accessed by PubMed), IEEE Xplore, Scopus, JMIR, and ACM Digital Library. Keywords used were (mental health OR mental illness OR mental disorder OR psychiatry) AND (large language models). This study included articles published between January 1, 2017, and April 30, 2024, and excluded articles published in languages other than English. RESULTS In total, 40 articles were evaluated, including 15 (38%) articles on mental health conditions and suicidal ideation detection through text analysis, 7 (18%) on the use of LLMs as mental health conversational agents, and 18 (45%) on other applications and evaluations of LLMs in mental health. LLMs show good effectiveness in detecting mental health issues and providing accessible, destigmatized eHealth services. However, assessments also indicate that the current risks associated with clinical use might surpass their benefits. These risks include inconsistencies in generated text; the production of hallucinations; and the absence of a comprehensive, benchmarked ethical framework. CONCLUSIONS This systematic review examines the clinical applications of LLMs in mental health, highlighting their potential and inherent risks. The study identifies several issues: the lack of multilingual datasets annotated by experts, concerns regarding the accuracy and reliability of generated content, challenges in interpretability due to the “black box” nature of LLMs, and ongoing ethical dilemmas. These ethical concerns include the absence of a clear, benchmarked ethical framework; data privacy issues; and the potential for overreliance on LLMs by both physicians and patients, which could compromise traditional medical practices. As a result, LLMs should not be considered substitutes for professional mental health services. However, the rapid development of LLMs underscores their potential as valuable clinical aids, emphasizing the need for continued research and development in this area. CLINICALTRIAL PROSPERO CRD42024508617; https://www.crd.york.ac.uk/prospero/display_record.php?RecordID=508617
164	doi:10.2196/52758	2024	Human-Comparable Sensitivity of Large Language Models in Identifying Eligible Studies Through Title and Abstract Screening: 3-Layer Strategy Using GPT-3.5 and GPT-4 for Systematic Reviews	Kentaro Matsui	Journal of Medical Internet Research	50	L1-openalex-llm-for-lit-search	1	SCREEN	exclude [E6] uses LLMs as a research subject, not as a review tool	BACKGROUND: The screening process for systematic reviews is resource-intensive. Although previous machine learning solutions have reported reductions in workload, they risked excluding relevant papers. OBJECTIVE: We evaluated the performance of a 3-layer screening method using GPT-3.5 and GPT-4 to streamline the title and abstract-screening process for systematic reviews. Our goal is to develop a screening method that maximizes sensitivity for identifying relevant records. METHODS: We conducted screenings on 2 of our previous systematic reviews related to the treatment of bipolar disorder, with 1381 records from the first review and 3146 from the second. Screenings were conducted using GPT-3.5 (gpt-3.5-turbo-0125) and GPT-4 (gpt-4-0125-preview) across three layers: (1) research design, (2) target patients, and (3) interventions and controls. The 3-layer screening was conducted using prompts tailored to each study. During this process, information extraction according to each study's inclusion criteria and optimization for screening were carried out using a GPT-4-based flow without manual adjustments. Records were evaluated at each layer, and those meeting the inclusion criteria at all layers were subsequently judged as included. RESULTS: On each layer, both GPT-3.5 and GPT-4 were able to process about 110 records per minute, and the total time required for screening the first and second studies was approximately 1 hour and 2 hours, respectively. In the first study, the sensitivities/specificities of the GPT-3.5 and GPT-4 were 0.900/0.709 and 0.806/0.996, respectively. Both screenings by GPT-3.5 and GPT-4 judged all 6 records used for the meta-analysis as included. In the second study, the sensitivities/specificities of the GPT-3.5 and GPT-4 were 0.958/0.116 and 0.875/0.855, respectively. The sensitivities for the relevant records align with those of human evaluators: 0.867-1.000 for the first study and 0.776-0.979 for the second study. Both screenings by GPT-3.5 and GPT-4 judged all 9 records used for the meta-analysis as included. After accounting for justifiably excluded records by GPT-4, the sensitivities/specificities of the GPT-4 screening were 0.962/0.996 in the first study and 0.943/0.855 in the second study. Further investigation indicated that the cases incorrectly excluded by GPT-3.5 were due to a lack of domain knowledge, while the cases incorrectly excluded by GPT-4 were due to misinterpretations of the inclusion criteria. CONCLUSIONS: Our 3-layer screening method with GPT-4 demonstrated acceptable level of sensitivity and specificity that supports its practical application in systematic review screenings. Future research should aim to generalize this approach and explore its effectiveness in diverse settings, both medical and nonmedical, to fully establish its use and operational feasibility.
165	doi:10.1007/978-3-031-63028-6_6	2024	Preliminary Systematic Review of Open-Source Large Language Models in Education	Michael Pin-Chuan Lin	Lecture notes in computer science	12	L1-openalex-llm-for-lit-search	1	SCREEN	exclude [E6] uses LLMs as a research subject, not as a review tool
166	doi:10.3352/jeehp.2024.21.6	2024	Opportunities, challenges, and future directions of large language models, including ChatGPT in medical education: a systematic scoping review	Xiaojun Xu	Journal of Educational Evaluation for Health Professions	165	L1-openalex-llm-for-lit-search	1	SCREEN	exclude [E6] uses LLMs as a research subject, not as a review tool	BACKGROUND: ChatGPT is a large language model (LLM) based on artificial intelligence (AI) capable of responding in multiple languages and generating nuanced and highly complex responses. While ChatGPT holds promising applications in medical education, its limitations and potential risks cannot be ignored. METHODS: A scoping review was conducted for English articles discussing ChatGPT in the context of medical education published after 2022. A literature search was performed using PubMed/MEDLINE, Embase, and Web of Science databases, and information was extracted from the relevant studies that were ultimately included. RESULTS: ChatGPT exhibits various potential applications in medical education, such as providing personalized learning plans and materials, creating clinical practice simulation scenarios, and assisting in writing articles. However, challenges associated with academic integrity, data accuracy, and potential harm to learning were also highlighted in the literature. The paper emphasizes certain recommendations for using ChatGPT, including the establishment of guidelines. Based on the review, 3 key research areas were proposed: cultivating the ability of medical students to use ChatGPT correctly, integrating ChatGPT into teaching activities and processes, and proposing standards for the use of AI by medical students. CONCLUSION: ChatGPT has the potential to transform medical education, but careful consideration is required for its full integration. To harness the full potential of ChatGPT in medical education, attention should not only be given to the capabilities of AI but also to its impact on students and teachers.
167	doi:10.31219/osf.io/wm2av	2024	Accelerating Systematic Reviews with Large Language Models: Current Practices and Recommendations	Bo HU		3	L1-openalex-llm-for-lit-search	1	SCREEN	exclude [E6] uses LLMs as a research subject, not as a review tool	This study investigates the application of Large Language Models (LLMs) in systematic reviews, emphasizing their performance, consistency, and potential for cost efficiency. Through a systematic lineage search, we included 76 studies from 734 articles. The findings reveal that LLMs demonstrate moderate to high performance in title and abstract screening, full-text screening, and data extraction. However, their performance is notably unstable in the stages of literature search and quality assessment. Prompt design emerges as a crucial factor, with Chain of Thought (CoT) prompts frequently enhancing results. While LLMs exhibit moderate to high agreement with human reviewers in some stages, their consistency in quality assessment remains comparatively lower. The research suggests that while LLMs cannot fully replace human reviewers, they serve as valuable assistants in systematic reviews, especially in reducing time and effort. The study also provides practical recommendations for integrating LLMs effectively and discusses the challenges and future research directions in this evolving field.
168	doi:10.1186/s13643-024-02575-4	2024	Title and abstract screening for literature reviews using large language models: an exploratory study in the biomedical domain	Fabio Dennstädt	Systematic Reviews	151	L1-openalex-llm-for-lit-search	1	SCREEN	maybe [LLM] LLM title/abstract screening (biomedical)	BACKGROUND: Systematically screening published literature to determine the relevant publications to synthesize in a review is a time-consuming and difficult task. Large language models (LLMs) are an emerging technology with promising capabilities for the automation of language-related tasks that may be useful for such a purpose. METHODS: LLMs were used as part of an automated system to evaluate the relevance of publications to a certain topic based on defined criteria and based on the title and abstract of each publication. A Python script was created to generate structured prompts consisting of text strings for instruction, title, abstract, and relevant criteria to be provided to an LLM. The relevance of a publication was evaluated by the LLM on a Likert scale (low relevance to high relevance). By specifying a threshold, different classifiers for inclusion/exclusion of publications could then be defined. The approach was used with four different openly available LLMs on ten published data sets of biomedical literature reviews and on a newly human-created data set for a hypothetical new systematic literature review. RESULTS: The performance of the classifiers varied depending on the LLM being used and on the data set analyzed. Regarding sensitivity/specificity, the classifiers yielded 94.48%/31.78% for the FlanT5 model, 97.58%/19.12% for the OpenHermes-NeuralChat model, 81.93%/75.19% for the Mixtral model and 97.58%/38.34% for the Platypus 2 model on the ten published data sets. The same classifiers yielded 100% sensitivity at a specificity of 12.58%, 4.54%, 62.47%, and 24.74% on the newly created data set. Changing the standard settings of the approach (minor adaption of instruction prompt and/or changing the range of the Likert scale from 1-5 to 1-10) had a considerable impact on the performance. CONCLUSIONS: LLMs can be used to evaluate the relevance of scientific publications to a certain review topic and classifiers based on such an approach show some promising results. To date, little is known about how well such systems would perform if used prospectively when conducting systematic literature reviews and what further implications this might have. However, it is likely that in the future researchers will increasingly use LLMs for evaluating and classifying scientific publications.
169	doi:10.1016/j.ajic.2024.03.016	2024	Utilizing natural language processing and large language models in the diagnosis and prediction of infectious diseases: A systematic review	Mahmud Omar	American Journal of Infection Control	59	L1-openalex-llm-for-lit-search	1	SCREEN	exclude [E6] uses LLMs as a research subject, not as a review tool
170	doi:10.1016/j.jval.2024.10.2293	2024	MSR59 Evaluating Systematic Literature Review Screening Performance of Seven Large Language Models in Response to Different Prompting Strategies	Ghayath Janoudi	Value in Health	1	L1-openalex-llm-for-lit-search	1	SCREEN	exclude [E6] uses LLMs as a research subject, not as a review tool
171	doi:10.1145/3695988	2024	Large Language Models for Software Engineering: A Systematic Literature Review	Xinyi Hou	ACM Transactions on Software Engineering and Methodology	1026	L2-openalex-llm-slr-software-eng	1	SCREEN	exclude [E6] uses LLMs as a research subject, not as a review tool	Large Language Models (LLMs) have significantly impacted numerous domains, including Software Engineering (SE). Many recent publications have explored LLMs applied to various SE tasks. Nevertheless, a comprehensive understanding of the application, effects, and possible limitations of LLMs on SE is still in its early stages. To bridge this gap, we conducted a Systematic Literature Review (SLR) on LLM4SE, with a particular focus on understanding how LLMs can be exploited to optimize processes and outcomes. We selected and analyzed 395 research articles from January 2017 to January 2024 to answer four key Research Questions (RQs). In RQ1, we categorize different LLMs that have been employed in SE tasks, characterizing their distinctive features and uses. In RQ2, we analyze the methods used in data collection, pre-processing, and application, highlighting the role of well-curated datasets for successful LLM for SE implementation. RQ3 investigates the strategies employed to optimize and evaluate the performance of LLMs in SE. Finally, RQ4 examines the specific SE tasks where LLMs have shown success to date, illustrating their practical contributions to the field. From the answers to these RQs, we discuss the current state-of-the-art and trends, identifying gaps in existing research, and highlighting promising areas for future study. Our artifacts are publicly available at https://github.com/security-pride/LLM4SE_SLR .
172	title:utilizinglargelanguagemodelstoupdatesystematicliteraturereviewsacasestudyontimepressureandwellbeinginsoftwareengineering	2024	Utilizing Large Language Models to Update Systematic Literature Reviews: A Case Study on Time Pressure and Well-Being in Software Engineering	Jingyi Guo	Työväentutkimus Vuosikirja	0	L2-openalex-llm-slr-software-eng	1	SCREEN	exclude [E6] uses LLMs as a research subject, not as a review tool	This thesis examines the application of Large Language Models (LLMs), including ChatGPT-4 and ChatGPT-4.o, to update systematic literature reviews (SLRs) with a focus on time pressure and well-being in software engineering. It aims to address the challenges of traditional SLRs, which often need more timeliness and automaticity in synthesizing the latest research findings. By conducting a case study, this research evaluates the efficiency, comprehensiveness, and accuracy of using LLMs to update SLRs, comparing the strengths and weaknesses of various LLMs in processing and analyzing vast amounts of literature. The study aims to provide insights into the potential of LLMs to revolutionize the methodology of SLRs, particularly in dynamically evolving fields such as software engineering.
173	doi:10.1145/3708525	2024	Efficient and Green Large Language Models for Software Engineering: Literature Review, Vision, and the Road Ahead	Jieke Shi	ACM Transactions on Software Engineering and Methodology	43	L2-openalex-llm-slr-software-eng	1	SCREEN	exclude [E6] uses LLMs as a research subject, not as a review tool	Large Language Models (LLMs) have recently shown remarkable capabilities in various software engineering tasks, spurring the rapid growth of the Large Language Models for Software Engineering (LLM4SE) area. However, limited attention has been paid to developing efficient LLM4SE techniques that demand minimal computational cost, time, and memory resources, as well as green LLM4SE solutions that reduce energy consumption, water usage, and carbon emissions. This article aims to redirect the focus of the research community toward the efficiency and greenness of LLM4SE, while also sharing potential research directions to achieve this goal. It commences with a brief overview of the significance of LLM4SE and highlights the need for efficient and green LLM4SE solutions. Subsequently, the article presents a vision for a future where efficient and green LLM4SE revolutionizes the LLM-based software engineering tool landscape, benefiting various stakeholders, including industry, individual practitioners, and society. The article then delineates a roadmap for future research, outlining specific research paths and potential solutions for the research community to pursue. While not intended to be a definitive guide, the article aims to inspire further progress, with the ultimate goal of establishing efficient and green LLM4SE as a central element in the future of software engineering.
174	doi:10.1145/3708522	2024	Large Language Model for Vulnerability Detection and Repair: Literature Review and the Road Ahead	Xin Zhou	ACM Transactions on Software Engineering and Methodology	137	L2-openalex-llm-slr-software-eng	1	SCREEN	exclude [E6] uses LLMs as a research subject, not as a review tool	The significant advancements in Large Language Models (LLMs) have resulted in their widespread adoption across various tasks within Software Engineering (SE), including vulnerability detection and repair. Numerous studies have investigated the application of LLMs to enhance vulnerability detection and repair tasks. Despite the increasing research interest, there is currently no existing survey that focuses on the utilization of LLMs for vulnerability detection and repair. In this paper, we aim to bridge this gap by offering a systematic literature review of approaches aimed at improving vulnerability detection and repair through the utilization of LLMs. The review encompasses research work from leading SE, AI, and Security conferences and journals, encompassing 43 papers published across 25 distinct venues, along with 15 high-quality preprint papers, bringing the total to 58 papers. By answering three key research questions, we aim to (1) summarize the LLMs employed in the relevant literature, (2) categorize various LLM adaptation techniques in vulnerability detection, and (3) classify various LLM adaptation techniques in vulnerability repair. Based on our findings, we have identified a series of limitations of existing studies. Additionally, we have outlined a roadmap highlighting potential opportunities that we believe are pertinent and crucial for future research endeavors.
175	doi:10.1109/compsac61105.2024.00096	2024	Systematic Literature Review of Prompt Engineering Patterns in Software Engineering	Yuya Sasaki	IEEE Annual International Computer Software and Applications Conference (COMPSAC)	23	L2-openalex-llm-slr-software-eng	1	SCREEN	exclude [E6] uses LLMs as a research subject, not as a review tool	Advancements in large language models (LLMs) are transforming software engineering through innovative prompt engineering strategies. By analyzing prompt-driven enhancements across key software engineering tasks, we present a sys-tematic literature review and a pioneering taxonomy elucidating the practical applications of prompt engineering in software engineering. Our taxonomy offers a foundational framework that clarifies the roles of prompt engineering and measures its impact, thereby guiding evolving AI -driven software engineering research and practices.
176	doi:10.1145/3674805.3686666	2024	ChatGPT application in Systematic Literature Reviews in Software Engineering: an evaluation of its accuracy to support the selection activity	Katia Romero Felizardo	Proceedings of the ... ACM-IEEE International Symposium on Empirical Software Engineering and Measurement/International Symposium on Empirical Software Engineering and Measurement	28	L2-openalex-llm-slr-software-eng	1	SCREEN	include [LLM] ChatGPT for SLR selection in SE: accuracy evaluation	Context: The Systematic Literature Review (SLR) process involves searching, selecting, and synthesizing relevant literature on a specific research topic for evidence-based decision-making in Software Engineering (SE). Due to the time-consuming of the SLR process, tool support is essential. Gap: ChatGPT is a significant advancement in Natural Language Processing (NLP), and it can potentially accelerate time-consuming and propone-error activities, such as the selection activity of the SLR process. Therefore, having a tool to assist in the selection process appears beneficial, and we argue that ChatGPT can facilitate the analysis of extensive studies, saving time and effort. Objective: We aim to evaluate the accuracy (i.e., studies correctly classified) of using ChatGPT–4.0 in SLR in SE, particularly to support the first stage, based on the title, abstract, and keywords. Method: We assessed the accuracy of utilizing ChatGPT for selecting studies, the first stage, to be included in two SLRs (SLR1 and SLR2), in contrast to the conventional method of reading the title and abstract. Results: The accuracy of ChatGPT supporting the initial selection activity was 75.3% (SLR1 – 101 correct selections: 48 inclusions and 53 exclusions; 33 incorrect selections: 17 inclusions and 16 exclusions) and 86.1% (SLR2 – 386 correct selections: 113 inclusions and 273 exclusions; 62 incorrect selections: 27 inclusions and 35 exclusions). Conclusions: Our accuracy results indicate that it is not advisable to completely outsource the selection process to ChatGPT. However, it could be valuable as a support tool, aiding novice researchers or even experienced ones when they are in doubt.
177	doi:10.1109/conisoft63288.2024.00050	2024	State of the Art of the Security of Code Generated by LLMs: A Systematic Literature Review	Leonardo Criollo Ramírez	International Conference in Software Engineering Research and Innovation (CONISOFT)	9	L2-openalex-llm-slr-software-eng	1	SCREEN	exclude [E6] uses LLMs as a research subject, not as a review tool	AI -assisted programming has experienced a surge in popularity over the past few years, largely thanks to advancements in Large Language Model technologies. This has led to the emergence of tools like ChatGPT and GitHub Copilot. However, the use of AI models for code generation comes with a downside: the resulting code is susceptible to vulnerabilities, thus posing new challenges in the field of secure software development. In this study, we analyze the current state of research regarding the security of LLM generated code from the Software Engineering perspective. We conducted a Systematic Literature Review following the guidelines from Kitchenham et al. The search process included five sources: IEEE Xplore, ACM, Science Direct, Springer Link and Wiley Online Library. We also included an iteration of backward and forward snowballing. We obtained 3104 peer-reviewed studies though Quasi-Gold aided automated search and selected the most relevant ones through 5 stages. The final selection includes 15 primary studies from which we extracted and synthesized data. We identified seven different kinds of security vulnerability present in LLM generated code, six different mitigation strategies and practices, and four tools recommended by authors to use in conjunction with LLM code generation. Security related issues within LLM generated code have only just begun to be explored, initial research has already emphasized the significance of considering the inclusion of AI-powered code generation in software projects, as it carries the risk of introducing vulnerabilities at a higher rate than human-generated code. The vulnerabilities, practices and tools identified in this study, can potentially help developers to use LLM programming assistants more responsibly, making informed decisions when leveraging LLM technology.
178	arxiv:2407.09577	2024	FlashNorm: Fast Normalization for Transformers	Nils Graef	arXiv.org	0	snowball-fwd-henry2020	1	SCREEN	exclude [E5] title/abstract: not about attention-logit control or LM training stability	Normalization layers are ubiquitous in large language models (LLMs) yet represent a compute bottleneck: on hardware with distinct vector and matrix execution units, the RMS calculation blocks the subsequent matrix multiplication, preventing parallel execution. We present FlashNorm, an exact reformulation of RMSNorm followed by a linear layer that (i) eliminates the normalization weights by folding them into the subsequent linear layer, and (ii) defers the scalar RMS normalization to the output of the matrix multiplication, enabling the two operations to execute in parallel. Additionally, by the scale invariance of RMS, an RMSNorm followed by a linear layer followed by another RMSNorm allows the first RMSNorm to be eliminated entirely -- a mathematically identical simplification that removes the pre-attention RMSNorm in models using QKV-normalization (e.g., Gemma~4) and in MLA-models with latent normalization (e.g., DeepSeek-V2, Mistral Small 4, and OpenMythos). The same techniques extend to LayerNorm, Dynamic Tanh (DyT), feed-forward networks with GLU variants, and RoPE-based attention. On an NVIDIA T4 GPU, FlashNorm achieves 33 - 35% lower latency on the norm-then-project operation in the compute-bound (prefill) regime at SmolLM2-135M scale, and 12 - 14% at Llama-7B scale. We verify zero-loss weight folding on three models. Beyond inference speed, FlashNorm simplifies model implementations by reducing parameter tensor count. Watch our explainer video https://youtu.be/GEuJv34_XgU?si and see https://github.com/OpenMachine-ai/transformer-tricks for code.
179	arxiv:2407.02258	2024	SiamTST: A Novel Representation Learning Framework for Enhanced Multivariate Time Series Forecasting applied to Telco Networks	S. Kristoffersen	PKDD/ECML Workshops	1	snowball-fwd-henry2020	1	SCREEN	exclude [E5] title/abstract: not about attention-logit control or LM training stability	We introduce SiamTST, a novel representation learning framework for multivariate time series. SiamTST integrates a Siamese network with attention, channel-independent patching, and normalization techniques to achieve superior performance. Evaluated on a real-world industrial telecommunication dataset, SiamTST demonstrates significant improvements in forecasting accuracy over existing methods. Notably, a simple linear network also shows competitive performance, achieving the second-best results, just behind SiamTST. The code is available at https://github.com/simenkristoff/SiamTST.
180	arxiv:2404.08801	2024	Megalodon: Efficient LLM Pretraining and Inference with Unlimited Context Length	Xuezhe Ma	Neural Information Processing Systems	63	snowball-fwd-henry2020	1	SCREEN	exclude [E5] title/abstract: not about attention-logit control or LM training stability	The quadratic complexity and weak length extrapolation of Transformers limits their ability to scale to long sequences, and while sub-quadratic solutions like linear attention and state space models exist, they empirically underperform Transformers in pretraining efficiency and downstream task accuracy. We introduce Megalodon, a neural architecture for efficient sequence modeling with unlimited context length. Megalodon inherits the architecture of Mega (exponential moving average with gated attention), and further introduces multiple technical components to improve its capability and stability, including complex exponential moving average (CEMA), timestep normalization layer, normalized attention mechanism and pre-norm with two-hop residual configuration. In a controlled head-to-head comparison with Llama2, Megalodon achieves better efficiency than Transformer in the scale of 7 billion parameters and 2 trillion training tokens. Megalodon reaches a training loss of 1.70, landing mid-way between Llama2-7B (1.75) and 13B (1.67). Code: https://github.com/XuezheMax/megalodon
181	arxiv:2410.13925	2024	FiTv2: Scalable and Improved Flexible Vision Transformer for Diffusion Model	Zidong Wang	arXiv.org	17	snowball-fwd-vit22b	1	SCREEN	exclude [E5] title/abstract: not about attention-logit control or LM training stability	\textit{Nature is infinitely resolution-free}. In the context of this reality, existing diffusion models, such as Diffusion Transformers, often face challenges when processing image resolutions outside of their trained domain. To address this limitation, we conceptualize images as sequences of tokens with dynamic sizes, rather than traditional methods that perceive images as fixed-resolution grids. This perspective enables a flexible training strategy that seamlessly accommodates various aspect ratios during both training and inference, thus promoting resolution generalization and eliminating biases introduced by image cropping. On this basis, we present the \textbf{Flexible Vision Transformer} (FiT), a transformer architecture specifically designed for generating images with \textit{unrestricted resolutions and aspect ratios}. We further upgrade the FiT to FiTv2 with several innovative designs, includingthe Query-Key vector normalization, the AdaLN-LoRA module, a rectified flow scheduler, and a Logit-Normal sampler. Enhanced by a meticulously adjusted network structure, FiTv2 exhibits $2\times$ convergence speed of FiT. When incorporating advanced training-free extrapolation techniques, FiTv2 demonstrates remarkable adaptability in both resolution extrapolation and diverse resolution generation. Additionally, our exploration of the scalability of the FiTv2 model reveals that larger models exhibit better computational efficiency. Furthermore, we introduce an efficient post-training strategy to adapt a pre-trained model for the high-resolution generation. Comprehensive experiments demonstrate the exceptional performance of FiTv2 across a broad range of resolutions. We have released all the codes and models at https://github.com/whlzy/FiT to promote the exploration of diffusion transformer models for arbitrary-resolution image generation.
182	arxiv:2409.18996	2024	From Linguistic Giants to Sensory Maestros: A Survey on Cross-Modal Reasoning with Large Language Models	Sheng-Sheng Qian	arXiv.org	7	snowball-fwd-vit22b	1	SCREEN	exclude [E5] title/abstract: not about attention-logit control or LM training stability	Cross-modal reasoning (CMR), the intricate process of synthesizing and drawing inferences across divergent sensory modalities, is increasingly recognized as a crucial capability in the progression toward more sophisticated and anthropomorphic artificial intelligence systems. Large Language Models (LLMs) represent a class of AI algorithms specifically engineered to parse, produce, and engage with human language on an extensive scale. The recent trend of deploying LLMs to tackle CMR tasks has marked a new mainstream of approaches for enhancing their effectiveness. This survey offers a nuanced exposition of current methodologies applied in CMR using LLMs, classifying these into a detailed three-tiered taxonomy. Moreover, the survey delves into the principal design strategies and operational techniques of prototypical models within this domain. Additionally, it articulates the prevailing challenges associated with the integration of LLMs in CMR and identifies prospective research directions. To sum up, this survey endeavors to expedite progress within this burgeoning field by endowing scholars with a holistic and detailed vista, showcasing the vanguard of current research whilst pinpointing potential avenues for advancement. An associated GitHub repository that collects the relevant papers can be found at https://github.com/ZuyiZhou/Awesome-Cross-modal-Reasoning-with-LLMs
183	arxiv:2402.17762	2024	Massive Activations in Large Language Models	Mingjie Sun	arXiv.org	332	snowball-fwd-wortsman2024	1	SCREEN	include [IM] massive activations -> attention sinks	We observe an empirical phenomenon in Large Language Models (LLMs) -- very few activations exhibit significantly larger values than others (e.g., 100,000 times larger). We call them massive activations. First, we demonstrate the widespread existence of massive activations across various LLMs and characterize their locations. Second, we find their values largely stay constant regardless of the input, and they function as indispensable bias terms in LLMs. Third, these massive activations lead to the concentration of attention probabilities to their corresponding tokens, and further, implicit bias terms in the self-attention output. Last, we also study massive activations in Vision Transformers. Code is available at https://github.com/locuslab/massive-activations.
184	arxiv:2410.13835	2024	Active-Dormant Attention Heads: Mechanistically Demystifying Extreme-Token Phenomena in LLMs	Tian-Yu Guo	arXiv.org	62	snowball-fwd-zhai2023	1	SCREEN	maybe [IM] extreme-token phenomena mechanism	Practitioners have consistently observed three puzzling phenomena in transformer-based large language models (LLMs): attention sinks, value-state drains, and residual-state peaks, collectively referred to as extreme-token phenomena. These phenomena are characterized by certain so-called"sink tokens"receiving disproportionately high attention weights, exhibiting significantly smaller value states, and having much larger residual-state norms than those of other tokens. These extreme tokens give rise to various challenges in LLM inference, quantization, and interpretability. We elucidate the mechanisms behind extreme-token phenomena. First, we show that these phenomena arise in very simple architectures -- transformers with one to three layers -- trained on a toy model, the Bigram-Backcopy (BB) task. In this setting, we identify an active-dormant mechanism, where attention heads become sinks for specific input domains while remaining non-sinks for others. Our theoretical analysis of the training dynamics reveals that these phenomena are driven by a mutual reinforcement mechanism. Building on these insights, we propose strategies to mitigate extreme-token phenomena during pretraining, including replacing softmax with ReLU and Adam with SGD. Next, we extend our analysis to pretrained LLMs, including Llama and OLMo, showing that many attention heads exhibit a similar active-dormant mechanism as in the BB task, and that the mutual reinforcement mechanism also governs the emergence of extreme-token phenomena during LLM pretraining. Our results reveal that many of the static and dynamic properties of extreme-token phenomena predicted by the BB task align with observations in pretrained LLMs.
185	arxiv:2410.09637	2024	ReLU's Revival: On the Entropic Overload in Normalization-Free Large Language Models	Nandan Kumar Jha	arXiv.org	4	snowball-fwd-zhai2023	1	SCREEN	exclude [E5] title/abstract: not about attention-logit control or LM training stability	LayerNorm is a critical component in modern large language models (LLMs) for stabilizing training and ensuring smooth optimization. However, it introduces significant challenges in mechanistic interpretability, outlier feature suppression, faithful signal propagation, and computational and communication complexity of private inference. This work explores desirable activation functions in normalization-free decoder-only LLMs. Contrary to the conventional preference for the GELU in transformer-based models, our empirical findings demonstrate an {\em opposite trend} -- ReLU significantly outperforms GELU in LayerNorm-free models, leading to an {\bf 8.2\%} perplexity improvement. We discover a key issue with GELU, where early layers experience entropic overload, leading to the under-utilization of the representational capacity of attention heads. This highlights that smoother activations like GELU are {\em ill-suited} for LayerNorm-free architectures, whereas ReLU's geometrical properties -- specialization in input space and intra-class selectivity -- lead to improved learning dynamics and better information retention in the absence of LayerNorm. This study offers key insights for optimizing transformer architectures where LayerNorm introduces significant challenges. The code and implementation are available at https://github.com/Nandan91/relu-revival-normfree
186	arxiv:2410.05052	2024	Initialization of Large Language Models via Reparameterization to Mitigate Loss Spikes	Kosuke Nishida	Conference on Empirical Methods in Natural Language Processing	7	snowball-fwd-zhai2023	1	SCREEN	exclude [E5] title/abstract: not about attention-logit control or LM training stability	Loss spikes, a phenomenon in which the loss value diverges suddenly, is a fundamental issue in the pre-training of large language models. This paper supposes that the non-uniformity of the norm of the parameters is one of the causes of loss spikes. Here, in training of neural networks, the scale of the gradients is required to be kept constant throughout the layers to avoid the vanishing and exploding gradients problem. However, to meet these requirements in the Transformer model, the norm of the model parameters must be non-uniform, and thus, parameters whose norm is smaller are more sensitive to the parameter update. To address this issue, we propose a novel technique, weight scaling as reparameterization (WeSaR). WeSaR introduces a gate parameter per parameter matrix and adjusts it to the value satisfying the requirements. Because of the gate parameter, WeSaR sets the norm of the original parameters uniformly, which results in stable training. Experimental results with the Transformer decoders consisting of 130 million, 1.3 billion, and 13 billion parameters showed that WeSaR stabilizes and accelerates training and that it outperformed compared methods including popular initialization methods.
187	arxiv:2409.07787	2024	Stable Language Model Pre-training by Reducing Embedding Variability	Woo-Jin Chung	Conference on Empirical Methods in Natural Language Processing	8	snowball-fwd-zhai2023	1	SCREEN	exclude [E5] title/abstract: not about attention-logit control or LM training stability	Stable pre-training is essential for achieving better-performing language models. However, tracking pre-training stability is impractical due to high computational costs. We study Token Embedding Variability as a simple proxy to estimate pre-training stability. We theoretically and empirically demonstrate that Multi-head Low-Rank Attention acts as a fundamental approach to reducing instability. This is supported by empirical findings on variants on GPT-2, demonstrating improved stability and lower perplexities, even at deeper layer counts.
188	arxiv:2405.15903	2024	UnitNorm: Rethinking Normalization for Transformers in Time Series	Nan Huang	arXiv.org	6	snowball-fwd-zhai2023	1	SCREEN	exclude [E5] title/abstract: not about attention-logit control or LM training stability	Normalization techniques are crucial for enhancing Transformer models' performance and stability in time series analysis tasks, yet traditional methods like batch and layer normalization often lead to issues such as token shift, attention shift, and sparse attention. We propose UnitNorm, a novel approach that scales input vectors by their norms and modulates attention patterns, effectively circumventing these challenges. Grounded in existing normalization frameworks, UnitNorm's effectiveness is demonstrated across diverse time series analysis tasks, including forecasting, classification, and anomaly detection, via a rigorous evaluation on 6 state-of-the-art models and 10 datasets. Notably, UnitNorm shows superior performance, especially in scenarios requiring robust attention mechanisms and contextual comprehension, evidenced by significant improvements by up to a 1.46 decrease in MSE for forecasting, and a 4.89% increase in accuracy for classification. This work not only calls for a reevaluation of normalization strategies in time series Transformers but also sets a new direction for enhancing model performance and stability. The source code is available at https://anonymous.4open.science/r/UnitNorm-5B84.
189	arxiv:2402.02098	2024	Self-attention Networks Localize When QK-eigenspectrum Concentrates	Han Bao	International Conference on Machine Learning	20	snowball-fwd-zhai2023	1	SCREEN	include [IM] QK eigenspectrum concentration and entropy collapse	The self-attention mechanism prevails in modern machine learning. It has an interesting functionality of adaptively selecting tokens from an input sequence by modulating the degree of attention localization, which many researchers speculate is the basis of the powerful model performance but complicates the underlying mechanism of the learning dynamics. In recent years, mainly two arguments have connected attention localization to the model performances. One is the rank collapse, where the embedded tokens by a self-attention block become very similar across different tokens, leading to a less expressive network. The other is the entropy collapse, where the attention probability approaches non-uniform and entails low entropy, making the learning dynamics more likely to be trapped in plateaus. These two failure modes may apparently contradict each other because the rank and entropy collapses are relevant to uniform and non-uniform attention, respectively. To this end, we characterize the notion of attention localization by the eigenspectrum of query-key parameter matrices and reveal that a small eigenspectrum variance leads attention to be localized. Interestingly, the small eigenspectrum variance prevents both rank and entropy collapse, leading to better model expressivity and trainability.
190	arxiv:2401.04301	2024	Setting the Record Straight on Transformer Oversmoothing	G. Dovonon	Trans. Mach. Learn. Res.	23	snowball-fwd-zhai2023	1	SCREEN	exclude [E5] title/abstract: not about attention-logit control or LM training stability	Transformer-based models have recently become wildly successful across a diverse set of domains. At the same time, recent work has shown empirically and theoretically that Transformers are inherently limited. Specifically, they argue that as model depth increases, Transformers oversmooth, i.e., inputs become more and more similar. A natural question is: How can Transformers achieve these successes given this shortcoming? In this work we test these observations empirically and theoretically and uncover a number of surprising findings. We find that there are cases where feature similarity increases but, contrary to prior results, this is not inevitable, even for existing pre-trained models. Theoretically, we show that smoothing behavior depends on the eigenspectrum of the value and projection weights. We verify this empirically and observe that the sign of layer normalization weights can influence this effect. Our analysis reveals a simple way to parameterize the weights of the Transformer update equations to influence smoothing behavior. We hope that our findings give ML researchers and practitioners additional insight into how to develop future Transformer-based models.
191	doi:10.1073/pnas.2411962122	2025	Transforming literature screening: The emerging role of large language models in systematic reviews	Fernando M. Delgado-Chaves	Proceedings of the National Academy of Sciences	86	L1-openalex-llm-for-lit-search	1	SCREEN	maybe [LLM] LLMs in literature screening (PNAS perspective)	Systematic reviews (SR) synthesize evidence-based medical literature, but they involve labor-intensive manual article screening. Large language models (LLMs) can select relevant literature, but their quality and efficacy are still being determined compared to humans. We evaluated the overlap between title- and abstract-based selected articles of 18 different LLMs and human-selected articles for three SR. In the three SRs, 185/4,662, 122/1,741, and 45/66 articles have been selected and considered for full-text screening by two independent reviewers. Due to technical variations and the inability of the LLMs to classify all records, the LLM's considered sample sizes were smaller. However, on average, the 18 LLMs classified 4,294 (min 4,130; max 4,329), 1,539 (min 1,449; max 1,574), and 27 (min 22; max 37) of the titles and abstracts correctly as either included or excluded for the three SRs, respectively. Additional analysis revealed that the definitions of the inclusion criteria and conceptual designs significantly influenced the LLM performances. In conclusion, LLMs can reduce one reviewer´s workload between 33% and 93% during title and abstract screening. However, the exact formulation of the inclusion and exclusion criteria should be refined beforehand for ideal support of the LLMs.
192	doi:10.1093/jamia/ocaf063	2025	The emergence of large language models as tools in literature reviews: a large language model-assisted systematic review	Dmitry Scherbakov	Journal of the American Medical Informatics Association	154	L1-openalex-llm-for-lit-search	1	SCREEN	maybe [LLM] LLMs as literature-review tools: review	OBJECTIVES: This study aims to summarize the usage of large language models (LLMs) in the process of creating a scientific review by looking at the methodological papers that describe the use of LLMs in review automation and the review papers that mention they were made with the support of LLMs. MATERIALS AND METHODS: The search was conducted in June 2024 in PubMed, Scopus, Dimensions, and Google Scholar by human reviewers. Screening and extraction process took place in Covidence with the help of LLM add-on based on the OpenAI GPT-4o model. ChatGPT and Scite.ai were used in cleaning the data, generating the code for figures, and drafting the manuscript. RESULTS: Of the 3788 articles retrieved, 172 studies were deemed eligible for the final review. ChatGPT and GPT-based LLM emerged as the most dominant architecture for review automation (n = 126, 73.2%). A significant number of review automation projects were found, but only a limited number of papers (n = 26, 15.1%) were actual reviews that acknowledged LLM usage. Most citations focused on the automation of a particular stage of review, such as Searching for publications (n = 60, 34.9%) and Data extraction (n = 54, 31.4%). When comparing the pooled performance of GPT-based and BERT-based models, the former was better in data extraction with a mean precision of 83.0% (SD = 10.4) and a recall of 86.0% (SD = 9.8). DISCUSSION AND CONCLUSION: Our LLM-assisted systematic review revealed a significant number of research projects related to review automation using LLMs. Despite limitations, such as lower accuracy of extraction for numeric data, we anticipate that LLMs will soon change the way scientific reviews are conducted.
193	doi:10.1136/rapm-2025-107204	2025	Beware of large language models for scoping literature searches	Ryan David Williams	Regional Anesthesia & Pain Medicine	0	L1-openalex-llm-for-lit-search	1	SCREEN	exclude [E6] uses LLMs as a research subject, not as a review tool	We read with interest the article by Dr Purwel and colleagues on the potential applications of large language models (LLMs) in meta-analyses and systematic reviews.[1][1] While we agree that LLMs could significantly reduce the time required for the laborious components of systematic reviews, we wish
194	doi:10.1038/s43856-024-00717-2	2025	Current applications and challenges in large language models for patient care: a systematic review	Felix Busch	Communications Medicine	355	L1-openalex-llm-for-lit-search	1	SCREEN	exclude [E6] uses LLMs as a research subject, not as a review tool	BACKGROUND: The introduction of large language models (LLMs) into clinical practice promises to improve patient education and empowerment, thereby personalizing medical care and broadening access to medical knowledge. Despite the popularity of LLMs, there is a significant gap in systematized information on their use in patient care. Therefore, this systematic review aims to synthesize current applications and limitations of LLMs in patient care. METHODS: We systematically searched 5 databases for qualitative, quantitative, and mixed methods articles on LLMs in patient care published between 2022 and 2023. From 4349 initial records, 89 studies across 29 medical specialties were included. Quality assessment was performed using the Mixed Methods Appraisal Tool 2018. A data-driven convergent synthesis approach was applied for thematic syntheses of LLM applications and limitations using free line-by-line coding in Dedoose. RESULTS: We show that most studies investigate Generative Pre-trained Transformers (GPT)-3.5 (53.2%, n = 66 of 124 different LLMs examined) and GPT-4 (26.6%, n = 33/124) in answering medical questions, followed by patient information generation, including medical text summarization or translation, and clinical documentation. Our analysis delineates two primary domains of LLM limitations: design and output. Design limitations include 6 second-order and 12 third-order codes, such as lack of medical domain optimization, data transparency, and accessibility issues, while output limitations include 9 second-order and 32 third-order codes, for example, non-reproducibility, non-comprehensiveness, incorrectness, unsafety, and bias. CONCLUSIONS: This review systematically maps LLM applications and limitations in patient care, providing a foundational framework and taxonomy for their implementation and evaluation in healthcare settings.
195	doi:10.7326/annals-24-02189	2025	Development of Prompt Templates for Large Language Model–Driven Screening in Systematic Reviews	Christian Cao	Annals of Internal Medicine	72	L1-openalex-llm-for-lit-search	1	SCREEN	exclude [E6] uses LLMs as a research subject, not as a review tool	BACKGROUND: Systematic reviews (SRs) are hindered by the initial rigorous article screen, which delays access to reliable information synthesis. OBJECTIVE: To develop generic prompt templates for large language model (LLM)-driven abstract and full-text screening that can be adapted to different reviews. DESIGN: Diagnostic test accuracy. SETTING: 48 425 citations were tested for abstract screening across 10 SRs. Full-text screening evaluated all 12 690 freely available articles from the original search. Prompt development used the GPT4-0125-preview model (OpenAI). PARTICIPANTS: None. MEASUREMENTS: Large language models were prompted to include or exclude articles based on SR eligibility criteria. Model outputs were compared with original SR author decisions after full-text screening to evaluate performance (accuracy, sensitivity, and specificity). RESULTS: Optimized prompts using GPT4-0125-preview achieved a weighted sensitivity of 97.7% (range, 86.7% to 100%) and specificity of 85.2% (range, 68.3% to 95.9%) in abstract screening and weighted sensitivity of 96.5% (range, 89.7% to 100.0%) and specificity of 91.2% (range, 80.7% to 100%) in full-text screening across 10 SRs. In contrast, zero-shot prompts had poor sensitivity (49.0% abstract, 49.1% full-text). Across LLMs, Claude-3.5 (Anthropic) and GPT4 variants had similar performance, whereas Gemini Pro (Google) and GPT3.5 (OpenAI) models underperformed. Direct screening costs for 10 000 citations differed substantially: Where single human abstract screening was estimated to require more than 83 hours and $1666.67 USD, our LLM-based approach completed screening in under 1 day for $157.02 USD. LIMITATIONS: Further prompt optimizations may exist. Retrospective study. Convenience sample of SRs. Full-text screening evaluations were limited to free PubMed Central full-text articles. CONCLUSION: A generic prompt for abstract and full-text screening achieving high sensitivity and specificity that can be adapted to other SRs and LLMs was developed. Our prompting innovations may have value to SR investigators and researchers conducting similar criteria-based tasks across the medical sciences. PRIMARY FUNDING SOURCE: None.
196	doi:10.5220/0013562900003967	2025	Evaluating Large Language Models for Literature Screening: A Systematic Review of Sensitivity and Workload Reduction	Elias Sandner	International Conference on Data Science, Technology and Applications (DATA)	4	L1-openalex-llm-for-lit-search	1	SCREEN	exclude [E6] uses LLMs as a research subject, not as a review tool
197	doi:10.2196/67488	2025	Accuracy of Large Language Models for Literature Screening in Thoracic Surgery: Diagnostic Study	Zhang-Yi Dai	Journal of Medical Internet Research	29	L1-openalex-llm-for-lit-search	1	SCREEN	exclude [E6] uses LLMs as a research subject, not as a review tool	BACKGROUND: Systematic reviews and meta-analyses rely on labor-intensive literature screening. While machine learning offers potential automation, its accuracy remains suboptimal. This raises the question of whether emerging large language models (LLMs) can provide a more accurate and efficient approach. OBJECTIVE: This paper evaluates the sensitivity, specificity, and summary receiver operating characteristic (SROC) curve of LLM-assisted literature screening. METHODS: We conducted a diagnostic study comparing the accuracy of LLM-assisted screening versus manual literature screening across 6 thoracic surgery meta-analyses. Manual screening by 2 investigators served as the reference standard. LLM-assisted screening was performed using ChatGPT-4o (OpenAI) and Claude-3.5 (Anthropic) sonnet, with discrepancies resolved by Gemini-1.5 pro (Google). In addition, 2 open-source, machine learning-based screening tools, ASReview (Utrecht University) and Abstrackr (Center for Evidence Synthesis in Health, Brown University School of Public Health), were also evaluated. We calculated sensitivity, specificity, and 95% CIs for the title and abstract, as well as full-text screening, generating pooled estimates and SROC curves. LLM prompts were revised based on a post hoc error analysis. RESULTS: LLM-assisted full-text screening demonstrated high pooled sensitivity (0.87, 95% CI 0.77-0.99) and specificity (0.96, 95% CI 0.91-0.98), with the area under the curve (AUC) of 0.96 (95% CI 0.94-0.97). Title and abstract screening achieved a pooled sensitivity of 0.73 (95% CI 0.57-0.85) and specificity of 0.99 (95% CI 0.97-0.99), with an AUC of 0.97 (95% CI 0.96-0.99). Post hoc revisions improved sensitivity to 0.98 (95% CI 0.74-1.00) while maintaining high specificity (0.98, 95% CI 0.94-0.99). In comparison, the pooled sensitivity and specificity of ASReview tool-assisted screening were 0.58 (95% CI 0.53-0.64) and 0.97 (95% CI 0.91-0.99), respectively, with an AUC of 0.66 (95% CI 0.62-0.70). The pooled sensitivity and specificity of Abstrackr tool-assisted screening were 0.48 (95% CI 0.35-0.62) and 0.96 (95% CI 0.88-0.99), respectively, with an AUC of 0.78 (95% CI 0.74-0.82). A post hoc meta-analysis revealed comparable effect sizes between LLM-assisted and conventional screening. CONCLUSIONS: LLMs hold significant potential for streamlining literature screening in systematic reviews, reducing workload without sacrificing quality. Importantly, LLMs outperformed traditional machine learning-based tools (ASReview and Abstrackr) in both sensitivity and AUC values, suggesting that LLMs offer a more accurate and efficient approach to literature screening.
198	doi:10.3390/biomedinformatics5020025	2025	Performance Comparison of Large Language Models for Efficient Literature Screening	Maria Teresa Colangelo	BioMedInformatics	6	L1-openalex-llm-for-lit-search	1	SCREEN	exclude [E6] uses LLMs as a research subject, not as a review tool	Background: Systematic reviewers face a growing body of biomedical literature, making early-stage article screening increasingly time-consuming. In this study, we assessed six large language models (LLMs)—OpenHermes, Flan T5, GPT-2, Claude 3 Haiku, GPT-3.5 Turbo, and GPT-4o—for their ability to identify randomized controlled trials (RCTs) in datasets of increasing difficulty. Methods: We first retrieved articles from PubMed and used all-mpnet-base-v2 to measure semantic similarity to known target RCTs, stratifying the collection into quartiles of descending relevance. Each LLM then received either verbose or concise prompts to classify articles as “Accepted” or “Rejected”. Results: Claude 3 Haiku, GPT-3.5 Turbo, and GPT-4o consistently achieved high recall, though their precision varied in the quartile with the highest similarity, where false positives increased. By contrast, smaller or older models struggled to balance sensitivity and specificity, with some over-including irrelevant studies or missing key articles. Importantly, multi-stage prompts did not guarantee performance gains for weaker models, whereas single-prompt approaches proved effective for advanced LLMs. Conclusions: These findings underscore that both model capability and prompt design strongly affect classification outcomes, suggesting that newer LLMs, if properly guided, can substantially expedite systematic reviews.
199	doi:10.1101/2025.06.13.25329541	2025	Automation of Systematic Reviews with Large Language Models	Christian Cao	medRxiv	33	L1-openalex-llm-for-lit-search	1	SCREEN	exclude [E6] uses LLMs as a research subject, not as a review tool	Abstract Importance Systematic reviews (SRs) inform evidence-based decision making. Yet, many take over a year to complete, are labor intensive, prone to human error, and face reproducibility challenges; thus limiting access to timely and reliable information. Objective To validate a large language model (LLM)-based workflow (otto-SR) to automate three of the most labour intensive tasks in performing SR’s: article screening, data extraction, and risk of bias assessment; and to assess its feasibility in rapidly updating existing reviews. Design, setting, and participants We conducted a validation study in four phases, with direct benchmarking against graduate-level human researchers in phases 1 and 2. Phase 1: article screening performance was measured across 32,357 citations from 5 systematic reviews. The reference standard consisted of the original reviews’ screening decisions after full-text screening. Phase 2: data extraction performance was measured across 4,495 data points from 495 studies in 7 reviews. Phase 3: risk of bias assessment (ROB2, Newcastle-Ottawa, QUADAS2) performance was measured across 345 studies from 12 reviews. Reference standards for Phase 2 and Phase 3 were created after blinded adjudication of the original review extraction and RoB assessments. Phase 4: otto-SR was used to reproduce and update the primary analysis from an issue of Cochrane reviews (n=12 reviews, 146,276 citations), with analytical comparisons to the original meta-analyzed findings. All discrepancies underwent dual human review. Results otto-SR showed high performance in phase 1 article screening ( otto-SR : 96.7% sensitivity, 97.9% specificity; human: 81.7% sensitivity, 98.1% specificity) and phase 2 data extraction ( otto-SR : 93.1% accuracy; human: 79.7% accuracy). In phase 3, otto-SR demonstrated high interrater reliability for risk of bias judgements (ROB2 0.98, Newcastle-Ottawa 0.95, QUADAS2 0.74; Gwet AC2). In phase 4, otto-SR , reproduced and updated the primary analysis from an issue of Cochrane reviews. Across Cochrane reviews, otto-SR incorrectly excluded a median of 0 studies (IQR 0 to 0.25), and found nearly twice as many eligible studies compared to the original authors (n= 114 vs. 64). Meta-analyses based on otto-SR generated screening and extraction outputs, subsequently verified through dual human review, yielded newly statistically significant effect estimates in 2 reviews and negated significance in 1 review. Conclusions and relevance LLMs have high performance in article screening, data extraction, and risk of bias assessments. They can rapidly reproduce and update existing systematic reviews, laying the foundation for automated, scalable, and reliable evidence synthesis.
200	doi:10.17605/osf.io/56d8j	2025	Performance of Large Language Models in Automated Medical Literature Screening: A Systematic Review and Meta-analysis	Chenggong Xie		0	L1-openalex-llm-for-lit-search	1	SCREEN	exclude [E6] uses LLMs as a research subject, not as a review tool	This project aims to systematically evaluate the performance, reliability, and heterogeneity of large language models (LLMs) in the automated screening of medical literature. As the volume of scientific publications continues to grow exponentially, the screening phase of systematic reviews has become increasingly time-consuming and labor-intensive. Recent advances in artificial intelligence—particularly large language models such as GPT-series, LLaMA, Claude, Gemini, Copilot, DeepSeek, and Qwen—have shown the potential to automate key steps of evidence synthesis. However, the accuracy, consistency, and generalizability of these models across different screening tasks and clinical domains remain unclear. This study will conduct a comprehensive systematic review and meta-analysis to summarize existing evidence on the use of LLMs for title/abstract screening, citation filtering, and study selection in medical and health-related systematic reviews. We will assess multiple performance indicators, including sensitivity, specificity, precision, recall, F1-score, workload reduction, and time efficiency. We also aim to explore sources of heterogeneity, such as model type, prompt design, clinical field, screening task complexity, dataset characteristics, and evaluation methodology. The review will follow PRISMA 2020 guidelines. Searches will be conducted across PubMed, Embase, Scopus, Web of Science, and preprint servers. Eligible studies will include evaluations of LLM-assisted or fully automated literature screening workflows, using either real-world systematic review datasets or curated benchmark sets. Risk of bias will be assessed using adapted tools for AI performance studies. Where feasible, meta-analysis will be performed to synthesize quantitative performance metrics.
201	doi:10.3390/info16050378	2025	Large Language Models in Systematic Review Screening: Opportunities, Challenges, and Methodological Considerations	Carlo Galli	Information	35	L1-openalex-llm-for-lit-search	1	SCREEN	exclude [E6] uses LLMs as a research subject, not as a review tool	Large Language Models (LLMs) have recently emerged as a powerful option for partially automating the labor-intensive process of screening articles in systematic reviews. Unlike traditional semi-automated platforms that rely on iterative human feedback, LLM-based pipelines can operate in a zero-shot or few-shot manner, classifying abstracts according to predefined criteria. This paper offers a step-by-step methodology for researchers, librarians, and students seeking to incorporate LLMs—such as GPT-4—into systematic reviews. It discusses required software and data preprocessing, presents various prompt strategies, and emphasizes the importance of human oversight to maintain rigorous quality control. The proposed framework aims to provide best practices and offer guidance on managing costs, reproducibility, and prompt refinement. By following these guidelines, review teams can substantially reduce screening workloads without compromising the comprehensive nature of evidence-based research.
202	doi:10.1016/j.jdent.2025.105877	2025	Large language models for the screening step in systematic reviews in dentistry	Rata Rokhshad	Journal of Dentistry	8	L1-openalex-llm-for-lit-search	1	SCREEN	exclude [E6] uses LLMs as a research subject, not as a review tool
203	doi:10.2196/76947	2025	Evaluating Large Language Models in Ophthalmology: Systematic Review	Zili Zhang	Journal of Medical Internet Research	34	L1-openalex-llm-for-lit-search	1	SCREEN	exclude [E6] uses LLMs as a research subject, not as a review tool	BACKGROUND Large language models (LLMs) have the potential to revolutionize ophthalmic care, but their evaluation practice remains fragmented. A systematic assessment is crucial to identify gaps and guide future evaluation practices and clinical integration. OBJECTIVE This study aims to map the current landscape of LLM evaluations in ophthalmology and explore whether performance synthesis is feasible for a common task. METHODS A comprehensive search of PubMed, Web of Science, Embase, and IEEE Xplore was conducted up to November 17, 2024 (no language limits). Eligible publications quantitatively assessed an existing or modified LLM on ophthalmology-related tasks. Studies without full-text availability or those focusing solely on vision-only models were excluded. Two reviewers screened studies and extracted data across 6 dimensions (evaluated LLM, data modality, ophthalmic subspecialty, medical task, evaluation dimension, and clinical alignment), and disagreements were resolved by a third reviewer. Descriptive statistics were analyzed and visualized using Python (with NumPy, Pandas, SciPy, and Matplotlib libraries). The Fisher exact test compared open- versus closed-source models. An exploratory random-effects meta-analysis (logit transformation; DerSimonian-Laird τ2) was performed for the diagnosis-making task; heterogeneity was reported with I2 and subgrouped by model, modality, and subspecialty. RESULTS Of the 817 identified records, 187 studies met the inclusion criteria. Closed-source LLMs dominated: 170 for ChatGPT, 58 for Gemini, and 32 for Copilot. Open-source LLMs appeared in only 25 (13.4%) of studies overall, but they appeared in 17 (77.3%) of evaluation-after-development studies, versus 8 (4.8%) pure-evaluation studies (P<1×10-5). Evaluations were chiefly text-only (n=168); image-text tasks, despite the centrality of imaging, were used in 19 studies. Subspecialty coverage was skewed toward comprehensive ophthalmology (n=72), retina and vitreous (n=32), and glaucoma (n=20). Refractive surgery, ocular pathology and oncology, and ophthalmic pharmacology each appeared in 3 or fewer studies. Medical query (n=86), standardized examination (n=41), and diagnosis making (n=29) emerged as the 3 predominant tasks, while research assistance (n=5), patient triaging (n=3), and disease prediction (n=3) received less attention. Accuracy was reported in most studies (n=176), whereas calibration and uncertainty were almost absent (n=5). Real-world patient data (n=45), human performance comparison (n=63), non‑English testing (n=24), and real-world deployment (n=4) were relatively absent. Exploratory meta-analysis pooled 28 diagnostic evaluations from 17 studies: overall accuracy was 0.594 (95% CI 0.488-0.692) with extreme heterogeneity (I2=94.5%). Subgroups remained heterogeneous (I2>80%), and findings were inconsistent (eg, pooled GPT-3.5>GPT-4). CONCLUSIONS Evidence on LLM evaluations in ophthalmology is extensive but heterogeneous. Most studies have tested a few closed-source LLMs on text-based questions, leaving open-source systems, multimodal tasks, non-English contexts, and real-world deployment underexamined. High methodological variability precludes meaningful performance aggregation, as illustrated by the heterogeneous meta-analysis. Standardized, multimodal benchmarks and phased clinical validation pipelines are urgently needed before LLMs can be safely integrated into eye care workflows.
204	doi:10.1016/j.bjoms.2025.03.006	2025	The impact of the large language model ChatGPT in oral and maxillofacial surgery: a systematic review	Lucas Mariz de Menezes Torres	British Journal of Oral and Maxillofacial Surgery	12	L1-openalex-llm-for-lit-search	1	SCREEN	exclude [E6] uses LLMs as a research subject, not as a review tool
205	doi:10.7326/annals-25-00012	2025	Large Language Models for Screening Search Results in Systematic Reviews: Are We There Yet?	S. Swaroop Vedula	Annals of Internal Medicine	1	L1-openalex-llm-for-lit-search	1	SCREEN	exclude [E6] uses LLMs as a research subject, not as a review tool
206	doi:10.1186/s12929-025-01131-z	2025	Impact of large language model (ChatGPT) in healthcare: an umbrella review and evidence synthesis	Usman Iqbal	Journal of Biomedical Science	133	L1-openalex-llm-for-lit-search	1	SCREEN	exclude [E6] uses LLMs as a research subject, not as a review tool	BACKGROUND: The emergence of Artificial Intelligence (AI), particularly Chat Generative Pre-Trained Transformer (ChatGPT), a Large Language Model (LLM), in healthcare promises to reshape patient care, clinical decision-making, and medical education. This review aims to synthesise research findings to consolidate the implications of ChatGPT integration in healthcare and identify research gaps. MAIN BODY: The umbrella review was conducted following Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) guidelines. The Cochrane Library, PubMed, Scopus, Web of Science, and Google Scholar were searched from inception until February 2024. Due to the heterogeneity of the included studies, no quantitative analysis was performed. Instead, information was extracted, summarised, synthesised, and presented in a narrative form. Two reviewers undertook title, abstract, and full text screening independently. The methodological quality and overall rating of the included reviews were assessed using the A Measurement Tool to Assess systematic Reviews (AMSTAR-2) checklist. The review examined 17 studies, comprising 15 systematic reviews and 2 meta-analyses, on ChatGPT in healthcare, revealing diverse focuses. The AMSTAR-2 assessment identified 5 moderate and 12 low-quality reviews, with deficiencies like study design justification and funding source reporting. The most reported theme that emerged was ChatGPT's use in disease diagnosis or clinical decision-making. While 82.4% of studies focused on its general usage, 17.6% explored unique topics like its role in medical examinations and conducting systematic reviews. Among these, 52.9% targeted general healthcare, with 41.2% focusing on specific domains like radiology, neurosurgery, gastroenterology, public health dentistry, and ophthalmology. ChatGPT's use for manuscript review or writing was mentioned in 17.6% of reviews. Promising applications include enhancing patient care and clinical decision-making, though ethical, legal, and accuracy concerns require cautious integration. CONCLUSION: We summarise the identified areas in reviews regarding ChatGPT's transformative impact in healthcare, highlighting patient care, decision-making, and medical education. Emphasising the importance of ethical regulations and the involvement of policymakers, we urge further investigation to ensure the reliability of ChatGPT and to promote trust in healthcare and research.
207	doi:10.1007/s12630-025-02973-9	2025	The applications of ChatGPT and other large language models in anesthesiology and critical care: a systematic review	Nicolas Daccache	Canadian Journal of Anesthesia/Journal canadien d anesthésie	15	L1-openalex-llm-for-lit-search	1	SCREEN	exclude [E6] uses LLMs as a research subject, not as a review tool
208	doi:10.1016/j.future.2025.107829	2025	The role of Large Language Models in addressing IoT challenges: A systematic literature review	Gabriele De Vito	Future Generation Computer Systems	20	L1-openalex-llm-for-lit-search	1	SCREEN	exclude [E6] uses LLMs as a research subject, not as a review tool	The Internet of Things (IoT) has revolutionized various sectors by enabling devices to communicate and interact seamlessly. However, developing IoT applications has data management, security, and interoperability challenges. Large Language Models (LLMs) have shown promise in addressing these challenges due to their advanced language processing capabilities. This Systematic Literature Review assesses the role of LLMs in addressing IoT challenges, exploring the strategies, hardware, and software configurations used, and identifying directions for future research. We extensively searched databases like Scopus, IEEE Xplore, and ACM Digital Library, initially screening 1419 studies and identifying an additional 1167 through snowballing, ultimately focusing on 55 relevant papers. The findings reveal LLMs’ potential to address key IoT challenges such as security and scalability. However, they also highlight significant obstacles, including high computational demands and the complexities of training and tuning these models. Future research should aim to develop methods to reduce the computational requirements of LLMs, improve training datasets, simplify implementation processes, and explore the ethical and privacy implications of using LLMs in IoT applications.
209	doi:10.1093/ehjdh/ztaf028	2025	Applications of large language models in cardiovascular disease: a systematic review	José Ferreira Santos	European Heart Journal - Digital Health	33	L1-openalex-llm-for-lit-search	1	SCREEN	exclude [E6] uses LLMs as a research subject, not as a review tool	Cardiovascular disease (CVD) remains the leading cause of morbidity and mortality worldwide. Large language models (LLMs) offer potential solutions for enhancing patient education and supporting clinical decision-making. This study aimed to evaluate LLMs' applications in CVD and explore their current implementation, from prevention to treatment. Following the Preferred Reporting Items for Systematic Reviews and Meta-Analyses guidelines, this systematic review assessed LLM applications in CVD. A comprehensive PubMed search identified relevant studies. The review prioritized pragmatic and practical applications of LLMs. Key applications, benefits, and limitations of LLMs in CVD prevention were summarized. Thirty-five observational studies met the eligibility criteria. Of these, 54% addressed primary prevention and risk factor management, while 46% focused on established CVD. Commercial LLMs were evaluated in all but one study, with 91% (32 studies) assessing ChatGPT. The LLM applications were categorized as follows: 72% addressed patient education, 17% clinical decision support, and 11% both. In 68% of studies, the primary objective was to evaluate LLMs' performance in answering frequently asked patient questions, with results indicating accurate, comprehensive, and generally safe responses. However, occasional misinformation and hallucinated references were noted. Additional applications included patient guidance on CVD, first aid, and lifestyle recommendations. Large language models were assessed for medical questions, diagnostic support, and treatment recommendations in clinical decision support. Large language models hold significant potential in CVD prevention and treatment. Evidence supports their potential as an alternative source of information for addressing patients' questions about common CVD. However, further validation is needed for their application in individualized care, from diagnosis to treatment.
210	doi:10.1038/s41366-025-01992-2	2025	Large language models in obesity: a systematic review	Thanathip Suenghataiphorn	International Journal of Obesity	6	L1-openalex-llm-for-lit-search	1	SCREEN	exclude [E6] uses LLMs as a research subject, not as a review tool
211	doi:10.1016/j.jclinepi.2025.111746	2025	Large language models for conducting systematic reviews: on the rise, but not yet ready for use—a scoping review	Judith-Lisa Lieberum	Journal of Clinical Epidemiology	146	L1-openalex-llm-for-lit-search	1	SCREEN	include [LLM] scoping review: LLMs for SRs 'not yet ready'	BACKGROUND AND OBJECTIVES: Machine learning promises versatile help in the creation of systematic reviews (SRs). Recently, further developments in the form of large language models (LLMs) and their application in SR conduct attracted attention. We aimed at providing an overview of LLM applications in SR conduct in health research. METHODS: We systematically searched MEDLINE, Web of Science, IEEEXplore, ACM Digital Library, Europe PMC (preprints), Google Scholar, and conducted an additional hand search (last search: February 26, 2024). We included scientific articles in English or German, published from April 2021 onwards, building upon the results of a mapping review that has not yet identified LLM applications to support SRs. Two reviewers independently screened studies for eligibility; after piloting, 1 reviewer extracted data, checked by another. RESULTS: Our database search yielded 8054 hits, and we identified 33 articles from our hand search. We finally included 37 articles on LLM support. LLM approaches covered 10 of 13 defined SR steps, most frequently literature search (n = 15, 41%), study selection (n = 14, 38%), and data extraction (n = 11, 30%). The mostly recurring LLM was Generative Pretrained Transformer (GPT) (n = 33, 89%). Validation studies were predominant (n = 21, 57%). In half of the studies, authors evaluated LLM use as promising (n = 20, 54%), one-quarter as neutral (n = 9, 24%) and one-fifth as nonpromising (n = 8, 22%). CONCLUSION: Although LLMs show promise in supporting SR creation, fully established or validated applications are often lacking. The rapid increase in research on LLMs for evidence synthesis production highlights their growing relevance. PLAIN LANGUAGE SUMMARY: Systematic reviews are a crucial tool in health research where experts carefully collect and analyze all available evidence on a specific research question. Creating these reviews is typically time- and resource-intensive, often taking months or even years to complete, as researchers must thoroughly search, evaluate, and synthesize an immense number of scientific studies. For the present article, we conducted a review to understand how new artificial intelligence (AI) tools, specifically large language models (LLMs) like Generative Pretrained Transformer (GPT), can be used to help create systematic reviews in health research. We searched multiple scientific databases and finally found 37 relevant articles. We found that LLMs have been tested to help with various parts of the systematic review process, particularly in 3 main areas: searching scientific literature (41% of studies), selecting relevant studies (38%), and extracting important information from these studies (30%). GPT was the most commonly used LLM, appearing in 89% of the studies. Most of the research (57%) focused on testing whether these AI tools actually work as intended in this context of systematic review production. The results were mixed: about half of the studies found LLMs promising, a quarter were neutral, and one-fifth found them not promising. While LLMs show potential for making the systematic review process more efficient, there is still a lack of fully tested and validated applications. However, the increasing number of studies in this field suggests that these AI tools are becoming increasingly important in creating systematic reviews.
212	doi:10.1093/dmfr/twaf060	2025	The performance of large language models in dentomaxillofacial radiology: a systematic review	Zekai Liu	Dentomaxillofacial Radiology	26	L1-openalex-llm-for-lit-search	1	SCREEN	exclude [E6] uses LLMs as a research subject, not as a review tool	OBJECTIVES: This study aimed to systematically review the current performance of large language models (LLMs) in dento-maxillofacial radiology (DMFR). METHODS: Five electronic databases were used to identify studies that developed, fine-tuned, or evaluated LLMs for DMFR-related tasks. Data extracted included study purpose, LLM type, images/text source, applied language, dataset characteristics, input and output, performance outcomes, evaluation methods, and reference standards. Customized assessment criteria adapted from the TRIPOD-LLM reporting guideline were used to evaluate the risk-of-bias in the included studies specifically regarding the clarity of dataset origin, the robustness of performance evaluation methods, and the validity of the reference standards. RESULTS: The initial search yielded 1621 titles, and 19 studies were included. These studies investigated the use of LLMs for tasks including the production and answering of DMFR-related qualification exams and educational questions (n = 8), diagnosis and treatment recommendations (n = 7), and radiology report generation and patient communication (n = 4). LLMs demonstrated varied performance in diagnosing dental conditions, with accuracy ranging from 37% to 92.5% and expert ratings for differential diagnosis and treatment planning between 3.6 and 4.7 on a 5-point scale. For DMFR-related qualification exams and board-style questions, LLMs achieved correctness rates between 33.3% and 86.1%. Automated radiology report generation showed moderate performance with accuracy ranging from 70.4% to 81.3%. CONCLUSIONS: LLMs demonstrate promising potential in DMFR, particularly for diagnostic, educational, and report generation tasks. However, their current accuracy, completeness, and consistency remain variable. Further development, validation, and standardization are needed before LLMs can be reliably integrated as supportive tools in clinical workflows and educational settings.
213	doi:10.2196/74177	2025	Large Language Models in Lung Cancer: Systematic Review	Ruikang Zhong	Journal of Medical Internet Research	9	L1-openalex-llm-for-lit-search	1	SCREEN	exclude [E6] uses LLMs as a research subject, not as a review tool	Background: In the era of data and intelligence, artificial intelligence has been widely applied in the medical field. As the most cutting-edge technology, the large language model (LLM) has gained popularity due to its extraordinary ability to handle complex tasks and interactive features. Objective: This study aimed to systematically review current applications of LLMs in lung cancer (LC) care and evaluate their potential across the full-cycle management spectrum. Methods: Following PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) guidelines, we conducted a comprehensive literature search across 6 databases up to January 1, 2025. Studies were included if they satisfied the following criteria: (1) journal articles, conference papers, and preprints; (2) studies that reported the content of LLMs in LC; (3) including original data and LC-related data presented separately; and (4) studies published in English. The exclusion criteria were as follows: (1) books and book chapters, letters, reviews, conference proceedings; (2) studies that did not report the content of LLMs in LC; and (3) no original data, and LC-related data that are not presented separately. Studies were screened independently by 2 authors (SC and ZL) and assessed for quality using Quality Assessment of Diagnostic Accuracy Studies-2, Prediction Model Risk of Bias Assessment Tool, and Risk Of Bias in Non-randomized Studies - of Interventions tools, selected based on study type. Key data items extracted included model type, application scenario, prompt method, input and output format, outcome measures, and safety considerations. Data analysis was conducted using descriptive statistics. Results: Out of 706 studies screened, 28 were included (published between 2023 and 2024). The ability of LLMs to automatically extract medical records, popularize general knowledge about LC, and assist clinical diagnosis and treatment has been demonstrated through the systematic review, emerging visual ability, and multimodal potential. Prompt engineering was a critical component, with varying degrees of sophistication from zero-shot to fine-tuned approaches. Quality assessments revealed overall acceptable methodological rigor but noted limitations in bias control and data security reporting. Conclusions: LLMs show considerable potential in improving LC diagnosis, communication, and decision-making. However, their responsible use requires attention to privacy, interpretability, and human oversight.
214	doi:10.71781/25708	2025	Automatic screening of articles for systematic reviews using large language models	Kumar, Gauransh		0	L1-openalex-llm-for-lit-search	1	SCREEN	exclude [E6] uses LLMs as a research subject, not as a review tool	Les revues systématiques de la littérature sont essentielles pour la recherche fondée sur des preuves mais exigent un travail manuel considérable, en particulier lors du tri des articles pour en déterminer la pertinence. Cette thèse explore comment les grands modèles de langage (Large Language Models, LLMs) peuvent aider à réduire cette charge tout en maintenant la qualité scientifique. Nous avons testé les LLMs comme assistants de sélection à laide de jeux de données variés et examiné limpact des différentes conceptions dinvite sur les performances. Notre étude a analysé la généralisabilité des invites entre différentes familles de LLMs, la relation entre la confiance du modèle et sa précision, ainsi que des stratégies pour combiner plusieurs approches. Nous avons développé PromptSLR, un outil permettant des tests contrôlés dinvites et une analyse de performance. Grâce à lélimination systématique des caractéristiques des invites, nous avons identifié des conceptions optimales à laide danalyses statistiques. Les résultats montrent que 2 invites atteignent les meilleures performances sur un ensemble de près de 30 invites, mais que leur efficacité varie fortement selon les modèles de LLM, avec une transférabilité limitée entre modèles. De façon critique, nous avons constaté que les LLMs excellent davantage à identifier les articles non pertinents quà sélectionner les pertinents, ce qui les rend idéaux pour le pré-tri plutôt que pour le tri principal. Il est observé que demander aux LLMs de sauto-évaluer (score de confiance) améliore leurs performances. Une confiance plus élevée du modèle corrèle avec une meilleure précision, permettant aux chercheurs de concentrer leffort manuel sur les cas incertains tout en automatisant lexclusion des évidences. Nous proposons une approche hybride combinant le pré-tri par LLM et le tri manuel traditionnel, ce qui réduit substantiellement la charge de travail des réviseurs tout en préservant la qualité du processus. Cette thèse positionne les LLMs comme des outils dassistance qui complètent lexpertise humaine plutôt que de la remplacer, et contribue à fournir des méthodes pratiques pour améliorer l'efficacité des SLRs ainsi que des outils pour une mise en oeuvre réelle.
215	doi:10.1017/rsm.2025.10014	2025	Optimal large language models to screen citations for systematic reviews	Takehiko Oami	Research Synthesis Methods	16	L1-openalex-llm-for-lit-search	1	SCREEN	exclude [E6] uses LLMs as a research subject, not as a review tool	Abstract Recent studies highlight the potential of large language models (LLMs) in citation screening for systematic reviews; however, the efficiency of individual LLMs for this application remains unclear. This study aimed to compare accuracy, time-related efficiency, cost, and consistency across four LLMs—GPT-4o, Gemini 1.5 Pro, Claude 3.5 Sonnet, and Llama 3.3 70B—for literature screening tasks. The models screened for clinical questions from the Japanese Clinical Practice Guidelines for the Management of Sepsis and Septic Shock 2024. Sensitivity and specificity were calculated for each model based on conventional citation screening results for qualitative assessment. We also recorded the time and cost of screening and assessed consistency to verify reproducibility. A post hoc analysis explored whether integrating outputs from multiple models could enhance screening accuracy. GPT-4o and Llama 3.3 70B achieved high specificity but lower sensitivity, while Gemini 1.5 Pro and Claude 3.5 Sonnet exhibited higher sensitivity at the cost of lower specificity. Citation screening times and costs varied, with GPT-4o being the fastest and Llama 3.3 70B the most cost-effective. Consistency was comparable among the models. An ensemble approach combining model outputs improved sensitivity but increased the number of false positives, requiring additional review effort. Each model demonstrated distinct strengths, effectively streamlining citation screening by saving time and reducing workload. However, reviewing false positives remains a challenge. Combining models may enhance sensitivity, indicating the potential of LLMs to optimize systematic review workflows.
216	doi:10.1016/j.acra.2025.08.014	2025	Evaluating Large Language Models for Radiology Systematic Review Title and Abstract Screening	Siddhant Dogra	Academic Radiology	6	L1-openalex-llm-for-lit-search	1	SCREEN	exclude [E6] uses LLMs as a research subject, not as a review tool
217	doi:10.1016/j.jclinane.2025.112030	2025	Combining large language models enhances screening sensitivity in systematic reviews	Alessandro DE CASSAI	Journal of Clinical Anesthesia	3	L1-openalex-llm-for-lit-search	1	SCREEN	exclude [E6] uses LLMs as a research subject, not as a review tool
218	doi:10.1017/rsm.2025.15	2025	Validation of large language models (Llama 3 and ChatGPT-4o mini) for title and abstract screening in biomedical systematic reviews	Adriana López‐Pineda	Research Synthesis Methods	17	L1-openalex-llm-for-lit-search	1	SCREEN	exclude [E6] uses LLMs as a research subject, not as a review tool	With the increasing volume of scientific literature, there is a need to streamline the screening process for titles and abstracts in systematic reviews, reduce the workload for reviewers, and minimize errors. This study validated artificial intelligence (AI) tools, specifically Llama 3 70B via Groq's application programming interface (API) and ChatGPT-4o mini via OpenAI's API, for automating this process in biomedical research. It compared these AI tools with human reviewers using 1,081 articles after duplicate removal. Each AI model was tested in three configurations to assess sensitivity, specificity, predictive values, and likelihood ratios. The Llama 3 model's LLA_2 configuration achieved 77.5% sensitivity and 91.4% specificity, with 90.2% accuracy, a positive predictive value (PPV) of 44.3%, and a negative predictive value (NPV) of 97.9%. The ChatGPT-4o mini model's CHAT_2 configuration showed 56.2% sensitivity, 95.1% specificity, 92.0% accuracy, a PPV of 50.6%, and an NPV of 96.1%. Both models demonstrated strong specificity, with CHAT_2 having higher overall accuracy. Despite these promising results, manual validation remains necessary to address false positives and negatives, ensuring that no important studies are overlooked. This study suggests that AI can significantly enhance efficiency and accuracy in systematic reviews, potentially revolutionizing not only biomedical research but also other fields requiring extensive literature reviews.
219	doi:10.1093/jamia/ocaf050	2025	High-performance automated abstract screening with large language model ensembles	Rohan Sanghera	Journal of the American Medical Informatics Association	55	L1-openalex-llm-for-lit-search	1	SCREEN	exclude [E6] uses LLMs as a research subject, not as a review tool	OBJECTIVE: screening is a labor-intensive component of systematic review involving repetitive application of inclusion and exclusion criteria on a large volume of studies. We aimed to validate large language models (LLMs) used to automate abstract screening. MATERIALS AND METHODS: LLMs (GPT-3.5 Turbo, GPT-4 Turbo, GPT-4o, Llama 3 70B, Gemini 1.5 Pro, and Claude Sonnet 3.5) were trialed across 23 Cochrane Library systematic reviews to evaluate their accuracy in zero-shot binary classification for abstract screening. Initial evaluation on a balanced development dataset (n = 800) identified optimal prompting strategies, and the best performing LLM-prompt combinations were then validated on a comprehensive dataset of replicated search results (n = 119 695). RESULTS: On the development dataset, LLMs exhibited superior performance to human researchers in terms of sensitivity (LLMmax = 1.000, humanmax = 0.775), precision (LLMmax = 0.927, humanmax = 0.911), and balanced accuracy (LLMmax = 0.904, humanmax = 0.865). When evaluated on the comprehensive dataset, the best performing LLM-prompt combinations exhibited consistent sensitivity (range 0.756-1.000) but diminished precision (range 0.004-0.096) due to class imbalance. In addition, 66 LLM-human and LLM-LLM ensembles exhibited perfect sensitivity with a maximal precision of 0.458 with the development dataset, decreasing to 0.1450 over the comprehensive dataset; but conferring workload reductions ranging between 37.55% and 99.11%. DISCUSSION: Automated abstract screening can reduce the screening workload in systematic review while maintaining quality. Performance variation between reviews highlights the importance of domain-specific validation before autonomous deployment. LLM-human ensembles can achieve similar benefits while maintaining human oversight over all records. CONCLUSION: LLMs may reduce the human labor cost of systematic review with maintained or improved accuracy, thereby increasing the efficiency and quality of evidence synthesis.
220	doi:10.3389/fdgth.2025.1659134	2025	Large language models in real-world clinical workflows: a systematic review of applications and implementation	Yaara Artsi	Frontiers in Digital Health	84	L1-openalex-llm-for-lit-search	1	SCREEN	exclude [E6] uses LLMs as a research subject, not as a review tool	Background: Large language models (LLMs) offer promise for enhancing clinical care by automating documentation, supporting decision-making, and improving communication. However, their integration into real-world healthcare workflows remains limited and under characterized. This systematic review aims to evaluate the literature on real-world implementation of LLMs in clinical workflows, including their use cases, clinical settings, observed outcomes, and challenges. Methods: We searched MEDLINE, Scopus, Web of Science, and Google Scholar for studies published between January 2015 and April 2025 that assessed LLMs in real-world clinical applications. Inclusion criteria were peer-reviewed, full-text studies in English reporting empirical implementation of LLMs in clinical settings. Study quality and risk of bias were assessed using the PROBAST tool. Results: Four studies published between 2024 and 2025 met inclusion criteria. All used generative pre-trained transformers (GPTs). Reported applications included outpatient communication, mental health support, inbox message drafting, and clinical data extraction. LLM deployment was associated with improvements in operational efficiency, user satisfaction, and reduced workload. However, challenges included performance variability across data types, limitations in generalizability, regulatory delays, and lack of post-deployment monitoring. Conclusions: Early evidence suggests that LLMs can enhance clinical workflows, but real-world adoption remains constrained by systemic, technical, and regulatory barriers. To support safe and scalable use, future efforts should prioritize standardized evaluation metrics, multi-site validation, human oversight, and implementation frameworks tailored to clinical settings. Systematic Review Registration: https://www.crd.york.ac.uk/PROSPERO/recorddashboard, PROSPERO CRD420251030069.
221	title:promptingstrategiesforlargelanguagemodelsinrequirementselicitationasystematicliteraturereview	2025	Prompting Strategies for Large Language Models in Requirements Elicitation : A Systematic Literature Review	Naveed Ali	Tampere University Institutional Repository (Tampere University)	0	L1-openalex-llm-for-lit-search	1	SCREEN	exclude [E6] uses LLMs as a research subject, not as a review tool	Recent advancements in Large Language Models (LLMs) such as GPT-4, DeepSeek, Gemini, and LLaMA have introduced new opportunities for supporting requirements elicitation, one of the most knowledge-intensive and error-prone activities in Requirements Engineering (RE). The effectiveness of LLM-assisted elicitation depends largely on how prompts are designed, structured, and contextualized. As a wide range of prompting strategies has emerged, including zero-shot, few-shot, chain-of-thought, role-based, and prompt-chaining techniques, there is a growing need to consolidate current evidence and evaluate their strengths, limitations, and practical relevance. This thesis presents a systematic literature review of 58 peer-reviewed studies investigating prompting strategies used with LLMs for requirements elicitation. The review follows established SLR guidelines and PRISMA 2020 procedures. Studies were retrieved from Scopus, IEEE Xplore, and the ACM Digital Library using a unified search strategy and underwent a structured screening, quality assessment, and data extraction process. The findings categorize prompting strategies across multiple dimensions, including their structural design, reasoning mechanisms, and integration with supporting tools such as retrieval-augmented generation pipelines, conversational agents, and domain-specific knowledge models. The review also synthesizes how prompting is applied across key elicitation contexts. Furthermore, it examines evaluation approaches, highlighting trends in quantitative metrics, expert judgment, and mixed-method assessment. The study contributes a consolidated view of prompting practices in LLM-based elicitation, identifies recurring benefits and challenges, and outlines emerging best practices. By synthesizing methodological patterns and gaps, this review offers guidance for researchers and practitioners seeking to design effective prompting strategies and develop reliable, human-centered elicitation workflows supported by generative AI.
222	doi:10.1097/hm9.0000000000000143	2025	Large language models in traditional Chinese medicine: a systematic review	Zhe Chen	Acupuncture and Herbal Medicine	16	L1-openalex-llm-for-lit-search	1	SCREEN	exclude [E6] uses LLMs as a research subject, not as a review tool	Objective: Generative artificial intelligence (AI) technology, represented by large language models (LLMs), has gradually been developed for traditional Chinese medicine (TCM); however, challenges remain in effectively enhancing AI applications for TCM. Therefore, this study is the first systematic review to analyze LLMs in TCM retrospectively, focusing on and summarizing the evidence of their performance in generative tasks. Methods: We extensively searched electronic databases for articles published until June 2024 to identify publicly available studies on LLMs in TCM. Two investigators independently selected and extracted the related information and evaluation metrics. Based on the available data, this study used descriptive analysis for a comprehensive systematic review of LLM technology related to TCM. Results: Ten studies published between 2023 and 2024 met our eligibility criteria and were included in this review, including 40% LLMs in the TCM vertical domain, 40% containing TCM data, and 20% honoring the TCM contribution, with a foundational model parameter range from 1.8 to 33 billion. All included studies used manual or automatic evaluation metrics to evaluate model performance and fully discussed the challenges and contributions through an overview of LLMs in TCM. Conclusions: LLMs have achieved significant advantages in TCM applications and can effectively address intelligent TCM tasks. Further in-depth development of LLMs is needed in various vertical TCM fields, including clinical and fundamental research. Focusing on the functional segmentation development direction of generative AI technologies in TCM application scenarios to meet the practical needs-oriented demands of TCM digitalization is essential.
223	doi:10.2196/preprints.76734	2025	Large Language Models in Cardiology: A Systematic Review (Preprint)	Moran Gendler		0	L1-openalex-llm-for-lit-search	1	SCREEN	exclude [E6] uses LLMs as a research subject, not as a review tool	UNSTRUCTURED Purpose: This review analyzes the application of large language models (LLMs) in the field of cardiology, with a focus on evaluating their performances across various clinical tasks. Methods: We conducted a systematic literature search on PubMed for studies published up to April 14, 2024. Our search strategy involved a combination of multiple keywords pertaining to LLMs and various aspects of cardiology. The risk of bias was evaluated using the QUADAS-2 tool. Results: Fifteen studies met the inclusion criteria, categorized into four domains: chronic and progressive cardiac conditions, acute cardiac events, cardiology education, and cardiac monitoring. Six studies addressing chronic conditions demonstrated variability in the accuracy and depth of LLM-generated responses. In acute cardiac scenarios, three articles showed that LLMs provided medical advice with mixed effectiveness, particularly in delivering CPR instructions. Two studies in educational cardiology revealed high accuracy in answering assessment questions and interpreting clinical cases. Finally, four articles on cardiac diagnostics showed that multimodal LLMs displayed a range of capabilities in ECGs interpretation, with some models performing at or exceeding the level of human specialists. Conclusion: LLMs demonstrate considerable potential in the field of cardiology, particularly in educational applications and routine diagnostics. However, their performance remains inconsistent across various clinical scenarios, particularly in acute care settings where precision is critical. Enhancing LLMs' accuracy in interpreting real-world complex medical data and emergency response guidance is imperative before integration into clinical practice.
224	doi:10.7717/peerj-cs.2822	2025	Validation of automated paper screening for esophagectomy systematic review using large language models	Rashi Ramchandani	PeerJ Computer Science	5	L1-openalex-llm-for-lit-search	1	SCREEN	exclude [E6] uses LLMs as a research subject, not as a review tool	Background: Large language models (LLMs) offer a potential solution to the labor-intensive nature of systematic reviews. This study evaluated the ability of the GPT model to identify articles that discuss perioperative risk factors for esophagectomy complications. To test the performance of the model, we tested GPT-4 on narrower inclusion criterion and by assessing its ability to discriminate relevant articles that solely identified preoperative risk factors for esophagectomy. Methods: = 1,967) discussing risk factors to esophagectomy complications. The articles underwent title and abstract screening by three independent human reviewers and GPT-4. The Python script used for the analysis made Application Programming Interface (API) calls to GPT-4 with screening criteria in natural language. GPT-4's inclusion and exclusion decision were compared to those decided human reviewers. Results: The agreement between the GPT model and human decision was 85.58% for perioperative factors and 78.75% for preoperative factors. The AUC value was 0.87 and 0.75 for the perioperative and preoperative risk factors query, respectively. In the evaluation of perioperative risk factors, the GPT model demonstrated a high recall for included studies at 89%, a positive predictive value of 74%, and a negative predictive value of 84%, with a low false positive rate of 6% and a macro-F1 score of 0.81. For preoperative risk factors, the model showed a recall of 67% for included studies, a positive predictive value of 65%, and a negative predictive value of 85%, with a false positive rate of 15% and a macro-F1 score of 0.66. The interobserver reliability was substantial, with a kappa score of 0.69 for perioperative factors and 0.61 for preoperative factors. Despite lower accuracy under more stringent criteria, the GPT model proved valuable in streamlining the systematic review workflow. Preliminary evaluation of inclusion and exclusion justification provided by the GPT model were reported to have been useful by study screeners, especially in resolving discrepancies during title and abstract screening. Conclusion: This study demonstrates promising use of LLMs to streamline the workflow of systematic reviews. The integration of LLMs in systematic reviews could lead to significant time and cost savings, however caution must be taken for reviews involving stringent a narrower and exclusion criterion. Future research is needed and should explore integrating LLMs in other steps of the systematic review, such as full text screening or data extraction, and compare different LLMs for their effectiveness in various types of systematic reviews.
225	doi:10.1038/s41746-025-01824-7	2025	Large language model integrations in cancer decision-making: a systematic review and meta-analysis	Yuexing Hao	npj Digital Medicine	66	L1-openalex-llm-for-lit-search	1	SCREEN	exclude [E6] uses LLMs as a research subject, not as a review tool	Large Language Models (LLMs) are increasingly used to support cancer patients and clinicians in decision-making. This systematic review investigates how LLMs are integrated into oncology and evaluated by researchers. We conducted a comprehensive search across PubMed, Web of Science, Scopus, and the ACM Digital Library through May 2024, identifying 56 studies covering 15 cancer types. The meta-analysis results suggested that LLMs were commonly used to summarize, translate, and communicate clinical information, but performance varied: the average overall accuracy was 76.2%, with average diagnostic accuracy lower at 67.4%, revealing gaps in the clinical readiness of this technology. Most evaluations relied heavily on quantitative datasets and automated methods without human graders, emphasizing "accuracy" and "appropriateness" while rarely addressing "safety", "harm", or "clarity". Current limitations for LLMs in cancer decision-making, such as limited domain knowledge and dependence on human oversight, demonstrate the need for open datasets and standardized evaluations to improve reliability.
226	doi:10.3390/app16010368	2025	Retrieval-Augmented Generation (RAG) and Large Language Models (LLMs) for Enterprise Knowledge Management and Document Automation: A Systematic Literature Review	Ehlullah Karakurt	Applied Sciences	25	L1-openalex-llm-for-lit-search	1	SCREEN	exclude [E6] uses LLMs as a research subject, not as a review tool	The integration of Retrieval-Augmented Generation (RAG) with Large Language Models (LLMs) is rapidly transforming enterprise knowledge management, yet a comprehensive understanding of their deployment in real-world workflows remains limited. This study presents a systematic literature review (SLR) analyzing 63 high-quality primary studies selected after rigorous screening to evaluate how these technologies address practical enterprise challenges. We formulated nine research questions targeting platforms, datasets, algorithms, and validation metrics to map the current landscape. Our findings reveal that enterprise adoption is largely in the experimental phase: 63.6% of implementations utilize GPT based models, and 80.5% rely on standard retrieval frameworks such as FAISS or Elasticsearch. Critically, this review identifies a significant ‘lab-to-market’ gap; while retrieval and classification sub-tasks frequently employ academic validation methods like k-fold cross-validation (93.6%), generative evaluation predominantly relies on static hold-out sets due to computational constraints. Furthermore, fewer than 15% of studies address real-time integration challenges required for production scale deployment. By systematically mapping these disparities, this study offers a data-driven perspective and a strategic roadmap for bridging the gap between academic prototypes and robust enterprise applications.
227	doi:10.1093/ckj/sfaf243	2025	Clinical applications and limitations of large language models in nephrology: a systematic review	Zoe Unger	Clinical Kidney Journal	12	L1-openalex-llm-for-lit-search	1	SCREEN	exclude [E6] uses LLMs as a research subject, not as a review tool	ABSTRACT Background Large language models (LLMs) have emerged as potential tools in healthcare. This systematic review evaluates the applications of text-generative conversational LLMs in nephrology, with particular attention to their reported advantages and limitations. Methods A systematic search was performed in PubMed, Web of Science, Embase and the Cochrane Library in accordance with the Preferred Reporting Items for Systematic Reviews and Meta-Analyses guidelines. Eligible studies assessed LLM applications in nephrology. PROSPERO registration number CRD42024550169. Results Of 1070 records screened, 23 studies met inclusion criteria, addressing four clinical applications in nephrology. In patient education (n = 13), GPT-4 improved the readability of kidney donation information from a 10th to a 4th grade level (9.6 ± 1.9 to 4.30 ± 1.71) and Gemini provided the most accurate answers to chronic kidney disease questions (Global Quality Score 3.46 ± 0.55). Regarding workflow optimization (n = 7), GPT-4 achieved high accuracy (90–94%) in managing continuous renal replacement therapy alarms and improved diagnosis of diabetes insipidus using chain-of-thought and retrieval-augmented prompting. In renal dietary guidance (n = 2), Bard AI led in classifying phosphorus and oxalate content of foods (100% and 84%), while GPT-4 and Bing Chat were most accurate for potassium classification (81%). For laboratory data interpretation (n = 1), Copilot significantly outperformed ChatGPT and Gemini in simulated nephrology datasets (median scores 5/5 compared with 4/5 and 4/5; P < .01). TRIPOD-LLM assessment revealed frequent omissions in data handling, prompting strategies and transparency. Conclusions While LLMs may enhance various aspects of nephrology practice, their widespread adoption remains premature. Input-quality dependence and limited external validation restrict generalizability. Further research is needed to confirm their real-world feasibility and ensure safe clinical integration.
228	arxiv:2502.15702	2025	Large language models streamline automated systematic review: A preliminary study	Xi Chen	arXiv (Cornell University)	2	L1-openalex-llm-for-lit-search	1	SCREEN	exclude [E6] uses LLMs as a research subject, not as a review tool	Large Language Models (LLMs) have shown promise in natural language processing tasks, with the potential to automate systematic reviews. This study evaluates the performance of three state-of-the-art LLMs in conducting systematic review tasks. We assessed GPT-4, Claude-3, and Mistral 8x7B across four systematic review tasks: study design formulation, search strategy development, literature screening, and data extraction. Sourced from a previously published systematic review, we provided reference standard including standard PICO (Population, Intervention, Comparison, Outcome) design, standard eligibility criteria, and data from 20 reference literature. Three investigators evaluated the quality of study design and eligibility criteria using 5-point Liker Scale in terms of accuracy, integrity, relevance, consistency and overall performance. For other tasks, the output is defined as accurate if it is the same as the reference standard. Search strategy performance was evaluated through accuracy and retrieval efficacy. Screening accuracy was assessed for both abstracts screening and full texts screening. Data extraction accuracy was evaluated across 1,120 data points comprising 3,360 individual fields. Claude-3 demonstrated superior overall performance in PICO design. In search strategy formulation, GPT-4 and Claude-3 achieved comparable accuracy, outperforming Mistral. For abstract screening, GPT-4 achieved the highest accuracy, followed by Mistral and Claude-3. In data extraction, GPT-4 significantly outperformed other models. LLMs demonstrate potential for automating systematic review tasks, with GPT-4 showing superior performance in search strategy formulation, literature screening and data extraction. These capabilities make them promising assistive tools for researchers and warrant further development and validation in this field.
229	doi:10.1016/j.ijmedinf.2025.106205	2025	AI-assisted literature screening: A hybrid approach using large language models and retrieval-augmented generation	Yiming Li	International Journal of Medical Informatics	8	L1-openalex-llm-for-lit-search	1	SCREEN	exclude [E6] uses LLMs as a research subject, not as a review tool
230	doi:10.1109/cinti67731.2025.11311831	2025	Large Language Models for Title/Abstract Screening in Systematic Literature Reviews: A Case Study in Precision Livestock Farming	Márk Zrubka	IEEE International Symposium on Computational Intelligence and Informatics (CINTI)	2	L1-openalex-llm-for-lit-search	1	SCREEN	exclude [E6] uses LLMs as a research subject, not as a review tool	Large language models (LLMs) are gaining popularity in systematic literature review automation, and they are even acknowledged as second reviewers along humans by the Cochrane Handbook. In this study, we tested the performance of two LLMs in the title/abstract screening process in a systematic review on precision livestock farming costbenefit analyses. We assessed the screening accuracy, sensitivity and specificity of the GPT-4o Mini and Claude 3.5 Sonnet models, as well as a human rater. We also compared the time and costs of completing the title/abstract screening by two human raters versus one human assisted by LLMs. LLMassisted screening required less time and money than two human raters, but it was also less accurate, resulting in extra work to rescreen discrepancies. Nevertheless, combining LLMs with the human ratings improved the quality of the screening output. Our results suggest that the tested LLMs are not as reliable as expert humans, however, they can assist in the title/abstract screening process and improve the quality of work with very low extra resources.
231	doi:10.3390/info16050420	2025	PROMPTHEUS: A Human-Centered Pipeline to Streamline Systematic Literature Reviews with Large Language Models	Joao Torres	Information	8	L1-openalex-llm-for-lit-search	1	SCREEN	exclude [E6] uses LLMs as a research subject, not as a review tool	The growing volume of academic publications poses significant challenges for researchers conducting timely and accurate systematic literature reviews (SLR), particularly in fast-evolving fields like artificial intelligence. This growth of academic literature also makes it increasingly difficult for lay people to access scientific knowledge effectively, meaning academic literature is often misrepresented in the popular press and, more broadly, in society. Traditional SLRs are labor-intensive and error-prone, and they struggle to keep up with the rapid pace of new research. To address these issues, we developed PROMPTHEUS: an AI-driven pipeline solution that automates the systematic literature review (LR) process using large language models (LLMs). We aimed to enhance efficiency by reducing the manual workload while maintaining the precision and coherence required for comprehensive literature synthesis. PROMPTHEUS automates key SLR stages, including systematic searches, data extraction, and topic modeling using BERTopic and summarization with transformer models. Evaluations across five research domains demonstrated that PROMPTHEUS reduces review time, achieves high precision, and provides coherent topic organization, offering a scalable and effective solution for conducting literature reviews in an increasingly crowded research landscape.
232	doi:10.1109/iccsm66818.2025.00027	2025	Toward the Conduction of Automated Systematic Literature Reviews: The Role of Large Language Models	Marco Fiore	International Conference on Computer, Software and Modeling (ICCSM)	0	L1-openalex-llm-for-lit-search	1	SCREEN	exclude [E6] uses LLMs as a research subject, not as a review tool	The conduction of Systematic Literature Reviews is an essential yet labor-intensive process for researchers, requiring meticulous compilation and analysis of literature to answer predefined Research Questions. Despite attempts to streamline this process, automation of reviews remains a significant challenge. This paper explores the potential of Generative AI and Large Language Models to revolutionize the review process by automating key tasks such as identifying, extracting, and synthesizing relevant literature. By leveraging open-source Large Language Models, we aim to reduce the time and effort traditionally required for comprehensive literature reviews. Our study involves comparing various models, including lightweight and more robust ones. We replicate a published review on Blockchain technology in agri-food traceability to evaluate the performance of these models. Results indicate that while high-parameter models successfully generate relevant search strings and classifications, they also exhibit trade-offs in terms of response times. Our findings highlight both the promise and limitations of current Large Language Models in automating Systematic Literature Reviews.
233	doi:10.1007/s10462-025-11398-1	2025	A systematic review of multi-modal large language models on domain-specific applications	Sirui Li	Artificial Intelligence Review	19	L1-openalex-llm-for-lit-search	1	SCREEN	exclude [E6] uses LLMs as a research subject, not as a review tool	Abstract While Large Language Models (LLMs) have shown remarkable proficiency in text-based tasks, they struggle to interact effectively with the more realistic world without the perceptions of other modalities such as visual and audio. Multi-modal LLMs, which integrate these additional modalities, have become increasingly important across various domains. Despite the significant advancements and potential of multi-modal LLMs, there has been no comprehensive PRISMA-based systematic review that examines their applications across different domains. The objective of this work is to fill this gap by systematically reviewing and synthesising the quantitative research literature on domain-specific applications of multi-modal LLMs. This systematic review follows the PRISMA guidelines to analyse research literature published after 2022, the release of OpenAI’s ChatGPT $$-$$ 3.5. The literature search was conducted across several online databases, including Nature, Scopus, and Google Scholar. A total of 22 studies were identified, with 11 focusing on the medical domain, 3 on autonomous driving, and 2 on geometric analysis. The remaining studies covered a range of topics, with one each on climate, music, e-commerce, sentiment analysis, human-robot interaction, and construction. This review provides a comprehensive overview of the current state of multi-modal LLMs, highlights their domain-specific applications, and identifies gaps and future research directions.
234	doi:10.1007/s10758-025-09922-x	2025	Large Language Model Applications in the Algebra Domain: A Systematic Review	Yajie Song	Technology Knowledge and Learning	0	L1-openalex-llm-for-lit-search	1	SCREEN	exclude [E6] uses LLMs as a research subject, not as a review tool
235	doi:10.2196/64963	2025	Comparing Diagnostic Accuracy of Clinical Professionals and Large Language Models: Systematic Review and Meta-Analysis	Guxue Shan	JMIR Medical Informatics	71	L1-openalex-llm-for-lit-search	1	SCREEN	exclude [E6] uses LLMs as a research subject, not as a review tool	Background: With the rapid development of artificial intelligence (AI) technology, especially generative AI, large language models (LLMs) have shown great potential in the medical field. Through massive medical data training, it can understand complex medical texts and can quickly analyze medical records and provide health counseling and diagnostic advice directly, especially in rare diseases. However, no study has yet compared and extensively discussed the diagnostic performance of LLMs with that of physicians. Objective: This study systematically reviewed the accuracy of LLMs in clinical diagnosis and provided reference for further clinical application. Methods: We conducted searches in CNKI (China National Knowledge Infrastructure), VIP Database, SinoMed, PubMed, Web of Science, Embase, and CINAHL (Cumulative Index to Nursing and Allied Health Literature) from January 1, 2017, to the present. A total of 2 reviewers independently screened the literature and extracted relevant information. The risk of bias was assessed using the Prediction Model Risk of Bias Assessment Tool (PROBAST), which evaluates both the risk of bias and the applicability of included studies. Results: A total of 30 studies involving 19 LLMs and a total of 4762 cases were included. The quality assessment indicated a high risk of bias in the majority of studies, primary cause is known case diagnosis. For the optimal model, the accuracy of the primary diagnosis ranged from 25% to 97.8%, while the triage accuracy ranged from 66.5% to 98%. Conclusions: LLMs have demonstrated considerable diagnostic capabilities and significant potential for application across various clinical cases. Although their accuracy still falls short of that of clinical professionals, if used cautiously, they have the potential to become one of the best intelligent assistants in the field of human health care.
236	doi:10.2196/preprints.89862	2025	Large Language Models in Colorectal Cancer: A Systematic Review (Preprint)	Jinglei Tian		0	L1-openalex-llm-for-lit-search	1	SCREEN	exclude [E6] uses LLMs as a research subject, not as a review tool	UNSTRUCTURED This systematic review aimed to analyze the application and efficacy of large language models (LLMs) in the full-cycle management of colorectal cancer (CRC). CRC diagnosis and treatment face challenges in multi-stage collaboration and integration of large-scale clinical data. While LLMs offer potential solutions, targeted evidence in CRC-specific settings remains insufficient. Following PRISMA guidelines, six databases including PubMed, Embase, and Web of Science were systematically searched up to November 1, 2025. Two reviewers independently screened studies, extracted data, and assessed quality using appropriate tools (QUADAS-2, PROBAST, ROBINS-I) based on study design. A total of 34 studies were included. LLMs were primarily applied in auxiliary diagnosis, information extraction, knowledge-based question answering, treatment decision support, and predictive modeling. GPT-4 achieved a diagnostic accuracy of 97.2% and an F1-score of 84.1% in information extraction, while lightweight or domain-specialized models (e.g., Gemma-2, RoBERTa) performed optimally in specific tasks. Significant heterogeneity was observed across studies due to variations in model types, prompt engineering strategies, and outcome measures. In conclusion, LLMs demonstrate considerable potential in supporting CRC management, particularly in data structuring and decision-making. However, limitations include data bias, model hallucination, and insufficient generalizability. Future research should focus on multicenter prospective validation, improvement of model interpretability, and development of multimodal and domain-adapted LLMs for safe and effective clinical integration.
237	doi:10.2196/preprints.74177	2025	Large Language Models in Lung Cancer: A Systematic Review (Preprint)	Ruikang Zhong		0	L1-openalex-llm-for-lit-search	1	SCREEN	exclude [E6] uses LLMs as a research subject, not as a review tool	BACKGROUND In the era of data and intelligence, Artificial Intelligence (AI), especially Large Language Model (LLM), has been widely applied in the medical field. OBJECTIVE The aim of this systematic review is to discuss the progress of LLM application in lung cancer, including clinical, educational and research aspects, and to explore the potential of LLM application in the full-cycle management of lung cancer. METHODS We searched six electronic databases according to PRISMA guidelines. Information was screened and extracted independently by two authors according to the inclusion criteria. Quality assessment was performed using the QUADAS-2, PROBAST and ROBINS-I tools. RESULTS The literature search yielded 706 relevant studies, resulting in the inclusion of a total of 28 studies published between 2023 and 2024.LLMs are widely used in lung cancer-related tasks, including assisted diagnosis, information extraction, question and answer science, and therapeutic decision support.ChatGPT is the most commonly used model and shows significant potential for improving diagnostic accuracy and patient communication. The importance of cue engineering and fine-tuning in optimising the performance of LLMs in specific clinical tasks is also highlighted. CONCLUSIONS LLMs have the potential to transform lung cancer management by improving diagnostic accuracy, patient communication and treatment planning. However, issues such as data security, ethical regulations, economic costs and clinical validation need to be addressed and LLMs need to be used rationally as effective tools rather than misused or even replaced by healthcare professionals.
238	doi:10.3233/shti251488	2025	Leveraging ChatGPT-4 for Evidence Synthesis: A Case Study on the Use of a Large Language Model in a Systematic Review	Federica Tomassini	Studies in health technology and informatics	0	L1-openalex-llm-for-lit-search	1	SCREEN	exclude [E6] uses LLMs as a research subject, not as a review tool	Artificial intelligence, particularly Large Language Models (LLM) such as ChatGPT, is emerging as a potentially transformative support for traditionally complex and time-consuming Systematic Literature Reviews (SLRs). In this study, we compared the traditional SLR process executed accordingly with Cochrane guidelines, with an AI-assisted approach using ChatGPT across various stages, from research question formulation to report writing. Effectiveness was assessed through quantitative measurements of time savings at each phase. Results showed substantial time reductions in several operational tasks, including Gantt chart, generating search terms and suggesting selection criteria. However, critical issues arose in stages requiring interpretative judgement, such as analyzing results, assessing risk of bias and final drafting. While AI cannot replace the role of the researcher, it is a valuable tool to optimize SLR workflow. The combination of human expertise and LLM capabilities presents a promising solution, provided it is accompanied by continuous development of AI systems to improve their reliability, transparency and interoperability.
239	doi:10.1002/cesm.70042	2025	Enhancing Evidence Synthesis Efficiency: Leveraging Large Language Models and Agentic Workflows for Optimized Literature Screening	Bing Hu	Cochrane Evidence Synthesis and Methods	4	L1-openalex-llm-for-lit-search	1	SCREEN	exclude [E6] uses LLMs as a research subject, not as a review tool	Background: Public health events of international concern highlight the need for up-to-date evidence curated using sustainable processes that are accessible. In development of the Global Repository of Epidemiological Parameters (grEPI) we explore the performance of an agentic-AI assisted pipeline (GREP-Agent) for screening evidence which capitalizes on recent advancements in large language models (LLMs). Methods: In this study, the performance of the GREP-Agent was evaluated on a data set of 2000 citations from a systematic review on measles using four LLMs (GPT4o, GPT4o-mini, Llama3.1, and Phi4). The GREP-Agent framework integrates multiple LLMs and human feedback to fine-tune its performance, optimize workload reduction and accuracy in screening research articles. The impact on performance of each part of this Agentic-AI system is presented and measured by accuracy, precision, recall, and F1-score metrics. Results: The results show how each phase of the GREP-Agent system incrementally improves accuracy regardless of the LLM. We found that GREP-Agent was able to increase sensitivity across a broad range of open source and proprietary LLMs to 84.2%-88.9% after fine-tuning and to 86.4%-95.3% by varying workload reduction strategies. Performance was significantly impacted by the clarity of the screening questions and setting thresholds for optimized workload reduction strategies. Conclusions: The GREP-Agent shows promise in improving the efficiency and effectiveness of evidence synthesis in dynamic public health contexts. Further development and refinement of adaptable human-in-the-loop AI systems for screening literature are essential to support future public health response activities, while maintaining a human-centric approach.
240	doi:10.1007/s00405-025-09504-8	2025	Clinical decision support using large language models in otolaryngology: a systematic review	Rania Filali Ansary	European Archives of Oto-Rhino-Laryngology	18	L1-openalex-llm-for-lit-search	1	SCREEN	exclude [E6] uses LLMs as a research subject, not as a review tool
241	title:analysisoftheeffectivenessoflargelanguagemodelsinindustrialknowledgemanagementasystematicliteraturereview	2025	Analysis of the Effectiveness of Large Language Models in Industrial Knowledge Management: A Systematic Literature Review	Jeronimo Malaquias	Jurnal Mekintek Jurnal Mekanikal Energi Industri Dan Teknologi	0	L1-openalex-llm-for-lit-search	1	SCREEN	exclude [E6] uses LLMs as a research subject, not as a review tool	The increasing adoption of Large Language Models (LLMs) has transformed the way organizations manage, access, and utilize knowledge in industrial environments. As industries continue to generate vast amounts of information, LLMs have emerged as powerful tools for enhancing knowledge management processes through advanced natural language understanding, information retrieval, and intelligent decision support capabilities. This study aims to analyze the effectiveness of LLMs in supporting industrial knowledge management and to evaluate their contributions, benefits, and challenges across organizational contexts. A Systematic Literature Review (SLR) approach was employed to examine relevant studies published between 2020 and 2026, with data collected from major scientific databases, including Scopus, Web of Science, IEEE Xplore, and ScienceDirect. The selected literature was analyzed using thematic, content, and comparative analysis techniques to identify patterns, applications, and implementation outcomes. The findings indicate that LLMs significantly enhance knowledge creation through automated report generation, technical documentation support, and lessons-learned extraction. Furthermore, LLM adoption contributes to increased organizational productivity by reducing information search time, supporting decision-making, and improving employee access to critical knowledge resources. However, several challenges remain, including hallucination, data inconsistency, model bias, integration complexity, security and privacy concerns, and issues related to transparency, accountability, and explainability. To maximize their benefits, organizations should implement robust AI governance frameworks, adopt secure knowledge retrieval architectures, and invest in employee AI literacy and training programs. Future research should focus on real-world industrial evaluations, comparative analyses of LLM platforms, and long-term assessments of organizational impacts.
242	doi:10.1016/j.jval.2025.04.1191	2025	MSR39 Large Language Model and Contextual Prioritization of Titles and Abstracts for Systematic Review Screening	Riaz Qureshi	Value in Health	0	L1-openalex-llm-for-lit-search	1	SCREEN	exclude [E6] uses LLMs as a research subject, not as a review tool
243	doi:10.1186/s12874-025-02583-5	2025	Streamlining systematic reviews with large language models using prompt engineering and retrieval augmented generation	Fouad S. Trad	BMC Medical Research Methodology	43	L1-openalex-llm-for-lit-search	1	SCREEN	exclude [E6] uses LLMs as a research subject, not as a review tool	BACKGROUND: Systematic reviews (SRs) are essential to formulate evidence-based guidelines but require time-consuming and costly literature screening. Large Language Models (LLMs) can be a powerful tool to expedite SRs. METHODS: We conducted a comparative study to evaluate the performance of a commercial tool, Rayyan, and an in-house LLM-based system in automating the screening of a completed SR on Vitamin D and falls. The SR retrieved 14,439 articles, and Rayyan was trained with 2,000 manually screened articles to categorize the rest as most likely to exclude/include, likely to exclude/include and undecided. We analyzed Rayyan's title/abstract screening performance using different inclusion thresholds. For the LLM, we used prompt engineering for title/abstract screening and Retrieval-Augmented Generation (RAG) for full-text screening. We evaluated performance using article exclusion rate (AER), false negative rate (FNR), specificity, positive predictive value (PPV), and negative predictive value (NPV). Additionally, we compared the time required to complete screening steps of the SR using both approaches against the manual screening method. RESULTS: Using Rayyan, including considered as undecided or likely to include for title/abstract screening resulted in an AER of 72.1% and an FNR of 5%. The total estimated screening time, including manual review of articles flagged by Rayyan, was 54.7 hours. Lowering the Rayyan threshold to 'likely to exclude' reduced the FNR to 0% and the AER to 50.7%, but increased the screening time to 81.3 h. Using the LLM system, after title/abstract and full-text screening, 78 articles remained for manual review, including all 20 identified by traditional methods. The LLM achieved an AER of 99.5%, specificity of 99.6%, PPV of 25.6%, and NPV of 100%, with a total screening time of 25.5 h, including manual review of the 78 articles, reducing the manual screening time by 95.5%. CONCLUSIONS: The LLM-based system significantly enhances SR efficiency, compared to manual methods and Rayyan while maintaining low FNR.
244	doi:10.2139/ssrn.5688905	2025	Systematic Review of Large Language Model Applications in Transport Engineering	Sneharsh Belsare	SSRN Electronic Journal	0	L1-openalex-llm-for-lit-search	1	SCREEN	exclude [E6] uses LLMs as a research subject, not as a review tool
245	doi:10.1016/j.jval.2025.09.2960	2025	SA18 Calibrating Large Language Model Probabilities to Achieve Target Recall in Systematic Review Screening	Ṣẹ̀yẹ Abògúnr̀in	Value in Health	0	L1-openalex-llm-for-lit-search	1	SCREEN	exclude [E6] uses LLMs as a research subject, not as a review tool
246	doi:10.3389/fdata.2024.1505284	2025	On explaining recommendations with Large Language Models: a review	Alan Said	Frontiers in Big Data	34	L1-openalex-llm-for-lit-search	1	SCREEN	exclude [E6] uses LLMs as a research subject, not as a review tool	The rise of Large Language Models (LLMs), such as LLaMA and ChatGPT, has opened new opportunities for enhancing recommender systems through improved explainability. This paper provides a systematic literature review focused on leveraging LLMs to generate explanations for recommendations-a critical aspect for fostering transparency and user trust. We conducted a comprehensive search within the ACM Guide to Computing Literature, covering publications from the launch of ChatGPT (November 2022) to the present (November 2024). Our search yielded 232 articles, but after applying inclusion criteria, only six were identified as directly addressing the use of LLMs in explaining recommendations. This scarcity highlights that, despite the rise of LLMs, their application in explainable recommender systems is still in an early stage. We analyze these select studies to understand current methodologies, identify challenges, and suggest directions for future research. Our findings underscore the potential of LLMs improving explanations of recommender systems and encourage the development of more transparent and user-centric recommendation explanation solutions.
247	doi:10.1093/jamia/ocaf008	2025	Improving large language model applications in biomedicine with retrieval-augmented generation: a systematic review, meta-analysis, and clinical development guidelines	Siru Liu	Journal of the American Medical Informatics Association	197	L1-openalex-llm-for-lit-search	1	SCREEN	exclude [E6] uses LLMs as a research subject, not as a review tool	OBJECTIVE: The objectives of this study are to synthesize findings from recent research of retrieval-augmented generation (RAG) and large language models (LLMs) in biomedicine and provide clinical development guidelines to improve effectiveness. MATERIALS AND METHODS: We conducted a systematic literature review and a meta-analysis. The report was created in adherence to the Preferred Reporting Items for Systematic Reviews and Meta-Analyses 2020 analysis. Searches were performed in 3 databases (PubMed, Embase, PsycINFO) using terms related to "retrieval augmented generation" and "large language model," for articles published in 2023 and 2024. We selected studies that compared baseline LLM performance with RAG performance. We developed a random-effect meta-analysis model, using odds ratio as the effect size. RESULTS: Among 335 studies, 20 were included in this literature review. The pooled effect size was 1.35, with a 95% confidence interval of 1.19-1.53, indicating a statistically significant effect (P = .001). We reported clinical tasks, baseline LLMs, retrieval sources and strategies, as well as evaluation methods. DISCUSSION: Building on our literature review, we developed Guidelines for Unified Implementation and Development of Enhanced LLM Applications with RAG in Clinical Settings to inform clinical applications using RAG. CONCLUSION: Overall, RAG implementation showed a 1.35 odds ratio increase in performance compared to baseline LLMs. Future research should focus on (1) system-level enhancement: the combination of RAG and agent, (2) knowledge-level enhancement: deep integration of knowledge into LLM, and (3) integration-level enhancement: integrating RAG systems within electronic health records.
248	doi:10.3389/fmed.2025.1670824	2025	Clinical applications of large language models in knee osteoarthritis: a systematic review	Zebing Ma	Frontiers in Medicine	6	L1-openalex-llm-for-lit-search	1	SCREEN	exclude [E6] uses LLMs as a research subject, not as a review tool	Background and aims: Knee osteoarthritis (KOA) is a common chronic degenerative disease that significantly impacts patients' quality of life. With the rapid advancement of artificial intelligence, large language models (LLMs) have demonstrated potential in supporting medical information extraction, clinical decision-making, and patient education through their natural language processing capabilities. However, the current landscape of LLM applications in the KOA domain, along with their methodological quality, has yet to be systematically reviewed. Therefore, this systematic review aims to comprehensively summarize existing clinical studies on LLMs in KOA, evaluate their performance and methodological rigor, and identify current challenges and future research directions. Methods: Following the PRISMA guidelines, a systematic search was conducted in PubMed, Cochrane Library, Embase databases and Web of science for literature published up to June 2025. The protocol was preregistered on the OSF platform. Studies were screened using standardized inclusion and exclusion criteria. Key study characteristics and performance evaluation metrics were extracted. Methodological quality was assessed using tools such as Cochrane RoB, STROBE, STARD, and DISCERN. Additionally, the CLEAR-LLM and CliMA-10 frameworks were applied to provide complementary evaluations of quality and performance. Results: A total of 16 studies were included, covering various LLMs such as ChatGPT, Gemini, and Claude. Application scenarios encompassed text generation, imaging diagnostics, and patient education. Most studies were observational in nature, and overall methodological quality ranged from moderate to high. Based on CliMA-10 scores, LLMs exhibited upper-moderate performance in KOA-related tasks. The ChatGPT-4 series consistently outperformed other models, especially in structured output generation, interpretation of clinical terminology, and content accuracy. Key limitations included insufficient sample representativeness, inconsistent control over hallucinated content, and the lack of standardized evaluation tools. Conclusion: Large language models show notable potential in the KOA field, but their clinical application is still exploratory and limited by issues such as sample bias and methodological heterogeneity. Model performance varies across tasks, underscoring the need for improved prompt design and standardized evaluation frameworks. With real-world data and ethical oversight, LLMs may contribute more significantly to personalized KOA management. Systematic review registration: https://osf.io/jy4kz, identifier 10.17605/OSF.IO/479R8.
249	doi:10.3389/fmed.2024.1512824	2025	Emerging applications of NLP and large language models in gastroenterology and hepatology: a systematic review	Mahmud Omar	Frontiers in Medicine	15	L1-openalex-llm-for-lit-search	1	SCREEN	exclude [E6] uses LLMs as a research subject, not as a review tool	Background and aim: In the last years, natural language processing (NLP) has transformed significantly with the introduction of large language models (LLM). This review updates on NLP and LLM applications and challenges in gastroenterology and hepatology. Methods: Registered with PROSPERO (CRD42024542275) and adhering to PRISMA guidelines, we searched six databases for relevant studies published from 2003 to 2024, ultimately including 57 studies. Results: Our review of 57 studies notes an increase in relevant publications in 2023-2024 compared to previous years, reflecting growing interest in newer models such as GPT-3 and GPT-4. The results demonstrate that NLP models have enhanced data extraction from electronic health records and other unstructured medical data sources. Key findings include high precision in identifying disease characteristics from unstructured reports and ongoing improvement in clinical decision-making. Risk of bias assessments using ROBINS-I, QUADAS-2, and PROBAST tools confirmed the methodological robustness of the included studies. Conclusion: NLP and LLMs can enhance diagnosis and treatment in gastroenterology and hepatology. They enable extraction of data from unstructured medical records, such as endoscopy reports and patient notes, and for enhancing clinical decision-making. Despite these advancements, integrating these tools into routine practice is still challenging. Future work should prospectively demonstrate real-world value.
250	doi:10.1089/lap.2024.0277	2025	The Role of Artificial Intelligence Large Language Models in Literature Search Assistance to Evaluate Inguinal Hernia Repair Approaches	João Pedro Gonçalves Kasakewitch	Journal of Laparoendoscopic & Advanced Surgical Techniques	3	L1-openalex-llm-for-lit-search	1	SCREEN	exclude [E6] uses LLMs as a research subject, not as a review tool	Aim: This study assesses the reliability of artificial intelligence (AI) large language models (LLMs) in identifying relevant literature comparing inguinal hernia repair techniques. Material and Methods: We used LLM chatbots (Bing Chat AI, ChatGPT versions 3.5 and 4.0, and Gemini) to find comparative studies and randomized controlled trials on inguinal hernia repair techniques. The results were then compared with existing systematic reviews (SRs) and meta-analyses and checked for the authenticity of listed articles. Results: LLMs screened 22 studies from 2006 to 2023 across eight journals, while the SRs encompassed a total of 42 studies. Through thorough external validation, 63.6% of the studies (14 out of 22), including 10 identified through Chat GPT 4.0 and 6 via Bing AI (with an overlap of 2 studies between them), were confirmed to be authentic. Conversely, 36.3% (8 out of 22) were revealed as fabrications by Google Gemini (Bard), with two (25.0%) of these fabrications mistakenly linked to valid DOIs. Four (25.6%) of the 14 real studies were acknowledged in the SRs, which represents 18.1% of all LLM-generated studies. LLMs missed a total of 38 (90.5%) of the studies included in the previous SRs, while 10 real studies were found by the LLMs but were not included in the previous SRs. Between those 10 studies, 6 were reviews, and 1 was published after the SRs, leaving a total of three comparative studies missed by the reviews. Conclusions: This study reveals the mixed reliability of AI language models in scientific searches. Emphasizing a cautious application of AI in academia and the importance of continuous evaluation of AI tools in scientific investigations.
251	arxiv:2504.02917	2025	Bias in Large Language Models Across Clinical Applications: A Systematic Review	Thanathip Suenghataiphorn	arXiv (Cornell University)	8	L1-openalex-llm-for-lit-search	1	SCREEN	exclude [E6] uses LLMs as a research subject, not as a review tool	Background: Large language models (LLMs) are rapidly being integrated into healthcare, promising to enhance various clinical tasks. However, concerns exist regarding their potential for bias, which could compromise patient care and exacerbate health inequities. This systematic review investigates the prevalence, sources, manifestations, and clinical implications of bias in LLMs. Methods: We conducted a systematic search of PubMed, OVID, and EMBASE from database inception through 2025, for studies evaluating bias in LLMs applied to clinical tasks. We extracted data on LLM type, bias source, bias manifestation, affected attributes, clinical task, evaluation methods, and outcomes. Risk of bias was assessed using a modified ROBINS-I tool. Results: Thirty-eight studies met inclusion criteria, revealing pervasive bias across various LLMs and clinical applications. Both data-related bias (from biased training data) and model-related bias (from model training) were significant contributors. Biases manifested as: allocative harm (e.g., differential treatment recommendations); representational harm (e.g., stereotypical associations, biased image generation); and performance disparities (e.g., variable output quality). These biases affected multiple attributes, most frequently race/ethnicity and gender, but also age, disability, and language. Conclusions: Bias in clinical LLMs is a pervasive and systemic issue, with a potential to lead to misdiagnosis and inappropriate treatment, particularly for marginalized patient populations. Rigorous evaluation of the model is crucial. Furthermore, the development and implementation of effective mitigation strategies, coupled with continuous monitoring in real-world clinical settings, are essential to ensure the safe, equitable, and trustworthy deployment of LLMs in healthcare.
252	doi:10.1136/bmjdhai-2025-000017	2025	Large language model enhanced framework for systematic reviews and meta-analyses	Jiashu Shen	BMJ Digital Health & AI	2	L1-openalex-llm-for-lit-search	1	SCREEN	exclude [E6] uses LLMs as a research subject, not as a review tool	Objective: To evaluate and synthesise current applications of large language models (LLMs) in systematic reviews and meta-analyses (SRMAs), identify key limitations and propose an enhanced theoretical framework to improve the efficiency, scalability and reliability of evidence synthesis. Methods and analysis: We conducted a narrative review of recent studies applying LLMs across key SRMA stages. A total of 21 publications were analysed for model type, task application, accuracy metrics and workflow impact. Building on this evidence base, we designed a comprehensive LLM-enhanced SRMA framework that categorises LLM roles as consultants and assistants, integrates human-in-the-loop strategies and uses retrieval-augmented generation (RAG) and agent-based architectures to address critical challenges including hallucinations, bias and workflow inefficiency. Results: The reviewed literature demonstrated that LLMs can support various SRMA tasks with reported accuracy ranging from 61% to 99%, showing particular promise in literature screening and data extraction. Our proposed framework conceptualises modular integration of LLMs across all six SRMA stages, with LLMs serving as consultants for research question formulation and search strategy development and as assistants for task automation including abstract screening and structured data extraction. The framework incorporates RAG technology to reduce hallucinations by grounding outputs in retrieved literature and employs agent-based orchestration for complex analytical workflows. Theoretical analysis suggests potential for significant efficiency gains while maintaining methodological rigour through strategic human oversight. Conclusion: LLMs offer substantial theoretical potential to transform evidence synthesis by improving efficiency, scalability and consistency across SRMA workflows. The proposed LLM-enhanced framework provides a systematic, theoretically grounded approach for integrating advanced artificial intelligence capabilities into existing SRMA methodologies while preserving essential human oversight and analytical integrity. Future empirical studies are needed to validate the framework's practical effectiveness, establish implementation protocols and demonstrate real-world benefits in evidence-based medicine.
253	doi:10.1101/2025.03.28.25324834	2025	Prompt Engineering in Large Language Models for Patient Education: A Systematic Review	Aya Mudrik	medRxiv	7	L1-openalex-llm-for-lit-search	1	SCREEN	exclude [E6] uses LLMs as a research subject, not as a review tool	ABSTRACT Background Large language models (LLMs) have shown promise in generating patient-friendly medical content, but their outputs often vary in accuracy, readability, and relevance. Prompt engineering—structuring inputs to guide LLM responses—may improve the quality of educational materials, yet its impact on patient education remains unclear. Objectives To systematically review whether prompt engineering improves readability, accuracy, and usability of LLM-generated content for patient education. Methods We conducted a systematic review in accordance with PRISMA guidelines. PubMed, Scopus, and Web of Science were searched for original studies evaluating prompt engineering techniques in patient education. Data were extracted on prompt types, LLM models used, and outcomes. Risk of bias was assessed using the QUADAS-2 tool, and a narrative synthesis was performed. Results Our search identified five studies that met our criteria, focusing on answering patient questions and generating medical information. Prompt engineering techniques included instruction-based, elaborated, role-defining, scene-defining, and domain-specific prompts. Structured prompting improved accuracy and comprehensiveness in several cases, particularly when specific formats or custom instructions were used. Readability gains were notable when prompts explicitly requested simpler language and reading levels, though some strategies unintentionally increased complexity. Variability in effectiveness across LLMs and prompt types was observed. Conclusion Prompt engineering can enhance the clarity and, in some cases, the accuracy of LLM-generated patient education materials. However, benefits vary by model and strategy. Standardized approaches and further research are needed to optimize prompts, minimize bias, and support reliable, accessible patient communication.
254	doi:10.1093/bib/bbaf631.059	2025	Should we use large language models to perform systematic reviews?	TH Chan	Briefings in Bioinformatics	0	L1-openalex-llm-for-lit-search	1	SCREEN	exclude [E6] uses LLMs as a research subject, not as a review tool	Abstract Background Large language models (LLMs) have shown promising performances in literature search, screening of full-text scientific articles, data extraction, and produce a written summary. LLMs have a strong potential in automating the process of conducting systematic reviews. Nonetheless, conducting systematic reviews is also a training exercise for critical appraisal of scientific evidence and synthesis of scientific claims. Overreliance on LLMs may erode comprehension and analytical skills in reading scientific articles over time, leading to cognitive offloading. Other risks may involve hallucination and responsibility problem. Methods To address both opportunities and limitations of LLMs when applied in systematic reviews, the LLM-Augmented Systematic Evidence Review (LASER) framework is proposed. Results The framework recommends different degree of LLM usage in different steps in the systematic review process to promote efficiency, ensure accuracy, and prevent cognitive offloading of the researcher involved. In this framework, the specific degree of LLM usage may depend on the career stage of the researcher and the accuracy of the LLM.
255	doi:10.3352/jeehp.2025.22.36	2025	Performance of large language models in medical licensing examinations: a systematic review and meta-analysis	Haniyeh Nouri	Journal of Educational Evaluation for Health Professions	14	L1-openalex-llm-for-lit-search	1	SCREEN	exclude [E6] uses LLMs as a research subject, not as a review tool	PURPOSE: This study systematically evaluates and compares the performance of large language models (LLMs) in answering medical licensing examination questions. By conducting subgroup analyses based on language, question format, and model type, this meta-analysis aims to provide a comprehensive overview of LLM capabilities in medical education and clinical decision-making. METHODS: This systematic review, registered in PROSPERO and following PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) guidelines, searched MEDLINE (PubMed), Scopus, and Web of Science for relevant articles published up to February 1, 2025. The search strategy included Medical Subject Headings (MeSH) terms and keywords related to ("ChatGPT" OR "GPT" OR "LLM variants") AND ("medical licensing exam" OR "medical exam" OR "medical education" OR "radiology exam*"). Eligible studies evaluated LLM accuracy on medical licensing examination questions. Pooled accuracy was estimated using a random-effects model, with subgroup analyses by LLM type, language, and question format. Publication bias was assessed using Egger's regression test. RESULTS: This systematic review identified 2,404 studies. After removing duplicates and excluding irrelevant articles through title and abstract screening, 36 studies were included after full-text review. The pooled accuracy was 72% (95% confidence interval, 70.0% to 75.0%) with high heterogeneity (I2=99%, P<0.001). Among LLMs, GPT-4 achieved the highest accuracy (81%), followed by Bing (79%), Claude (74%), Gemini/Bard (70%), and GPT-3.5 (60%) (P=0.001). Performance differences across languages (range, 62% in Polish to 77% in German) were not statistically significant (P=0.170). CONCLUSION: LLMs, particularly GPT-4, can match or exceed medical students' examination performance and may serve as supportive educational tools. However, due to variability and the risk of errors, they should be used cautiously as complements rather than replacements for traditional learning methods.
256	doi:10.2196/76252	2025	Automated Literature Screening for Hepatocellular Carcinoma Treatment Through Integration of 3 Large Language Models: Methodological Study	Chen Pan	JMIR Medical Informatics	6	L1-openalex-llm-for-lit-search	1	SCREEN	exclude [E6] uses LLMs as a research subject, not as a review tool	BACKGROUND Primary liver cancer, particularly hepatocellular carcinoma (HCC), poses significant clinical challenges due to late-stage diagnosis, tumor heterogeneity, and rapidly evolving therapeutic strategies. While systematic reviews and meta-analyses are essential for updating clinical guidelines, their labor-intensive nature limits timely evidence synthesis. OBJECTIVE This study proposes an automated literature screening workflow powered by large language models (LLMs) to accelerate evidence synthesis for HCC treatment guidelines. METHODS We developed a tripartite LLM framework integrating Doubao-1.5-pro-32k, Deepseek-v3, and DeepSeek-R1-Distill-Qwen-7B to simulate collaborative decision-making for study inclusion and exclusion. The system was evaluated across 9 reconstructed datasets derived from published HCC meta-analyses, with performance assessed using accuracy, agreement metrics (κ and prevalence-adjusted bias-adjusted κ), recall, precision, F1-scores, and computational efficiency parameters (processing time and cost). RESULTS The framework demonstrated good performance, with a weighted accuracy of 0.96 and substantial agreement (prevalence-adjusted bias-adjusted κ=0.91), achieving high weighted recall (0.90) but modest weighted precision (0.15) and F1-scores (0.22). Computational efficiency varied across datasets (processing time: 248-5850 s; cost: US $0.14-$3.68 per dataset). CONCLUSIONS This LLM-driven approach shows promise for accelerating evidence synthesis in HCC care by reducing screening time while maintaining methodological rigor. Key limitations related to clinical context sensitivity and error propagation highlight the need for reinforcement learning integration and domain-specific fine-tuning. LLM agent architectures with reinforcement learning offer a practical path for streamlining guideline updates, though further optimization is needed to improve specialization and reliability in complex clinical settings.
257	doi:10.1007/s10815-025-03679-z	2025	The potential of large language models in the field of infertility: a systematic review	Wei Li	Journal of Assisted Reproduction and Genetics	1	L1-openalex-llm-for-lit-search	1	SCREEN	exclude [E6] uses LLMs as a research subject, not as a review tool
258	doi:10.1016/j.identj.2025.04.015	2025	Can Large Language Models Serve as Reliable Tools for Information in Dentistry? A Systematic Review	Nora Alhazmi	International Dental Journal	24	L1-openalex-llm-for-lit-search	1	SCREEN	exclude [E6] uses LLMs as a research subject, not as a review tool	Large language models (LLMs) have gained popularity among dental students for generating subject-related answers. However, their widespread use raises significant concerns about misinformation. This systematic review aims to critically evaluate studies assessing the performance of LLMs in dentistry. A comprehensive electronic search was conducted in PubMed/Medline, Scopus, Embase, Web of Science, Google Scholar, and the Saudi Digital Library to identify studies published up to September 2024. The study quality was assessed using the Prediction Model Risk of Bias Assessment Tool (PROBAST). A total of 2030 studies have been identified. After removing 907 duplicate records, 1123 studies remained for screening. Ultimately, 31 studies met the inclusion criteria. Approximately half of these studies were classified as "high risk," while the remainder were classified as "low risk." The applicability of the findings was rated as "low concern." The primary limitations of LLMs include their inability to specify information sources and their tendency to generate fabricated citations. Based on this review, LLMs hold promise as supplementary educational tools in dentistry. Evidence indicates that students using LLMs may achieve improved academic performance compared to traditional methods. However, concerns about occasional inaccuracies and unreliable citations underscore the need for further research, integration with validated sources, and adherence to ethical guidelines. Ultimately, LLMs should be viewed as complementary tools within dental education, with careful consideration of their limitations.
259	doi:10.1200/cci-25-00076	2025	Mitigating Ethical Issues for Large Language Models in Oncology: A Systematic Review	Shuang Zhou	JCO Clinical Cancer Informatics	4	L1-openalex-llm-for-lit-search	1	SCREEN	exclude [E6] uses LLMs as a research subject, not as a review tool	PURPOSE: Large language models (LLMs) have demonstrated remarkable versatility in oncology applications, such as cancer staging and survival analysis. Despite their potential, ethical concerns such as data privacy breaches, bias in training data, lack of transparency, and risks associated with erroneous outputs pose significant challenges to their adoption in high-stakes oncology settings. Therefore, we aim to explore the ethical challenges associated with LLM-based applications in oncology and evaluate emerging techniques designed to address these issues. METHODS: Following the Preferred Reporting Items for Systematic Reviews and Meta-Analyses framework, a systematic review was conducted to evaluate publications related to ethical issues of LLMs in oncology across eight academic databases (eg, PubMed, Web of Science, and Embase) between January 1, 2019, and December 31, 2024. RESULTS: The search retrieved 4,319 published articles, of which 65 publications were preserved and included in our analysis. We identified six prevalent ethical challenges in oncology, including trust, equity, privacy, transparency, nonmaleficence, and accountability. We then evaluated emerging technical solutions to mitigate ethical challenges and summarized evaluation metrics used to assess these solutions' effectiveness. CONCLUSION: This review provides actionable recommendations for responsibly deploying LLMs in oncology, ensuring adherence to ethical guidelines, and fostering improved patient outcomes. By bridging technical and clinical perspectives, this review offers a foundational framework for advancing ethical artificial intelligence applications in oncology and highlights areas for future research.
260	doi:10.34133/hds.0322	2025	LitAutoScreener: Development and Validation of an Automated Literature Screening Tool in Evidence-Based Medicine Driven by Large Language Models	Yiming Tao	Health Data Science	9	L1-openalex-llm-for-lit-search	1	SCREEN	exclude [E6] uses LLMs as a research subject, not as a review tool	Background: The traditional manual literature screening approach is limited by its time-consuming nature and high labor costs. A pressing issue is how to leverage large language models to enhance the efficiency and quality of evidence-based evaluations of drug efficacy and safety. Methods: This study utilized a manually curated reference literature database—comprising vaccine, hypoglycemic agent, and antidepressant evaluation studies—previously developed by our team through conventional systematic review methods. This validated database served as the gold standard for the development and optimization of LitAutoScreener. Following the PICOS (Population, Intervention, Comparison, Outcomes, Study Design) principles, a chain-of-thought reasoning approach with few-shot learning prompts was implemented to develop the screening algorithm. We subsequently evaluated the performance of LitAutoScreener using 2 independent validation cohorts, assessing both classification accuracy and processing efficiency. Results: For respiratory syncytial virus vaccine safety validation title–abstract screening, our tools based on GPT (GPT-4o), Kimi (moonshot-v1-128k), and DeepSeek (deepseek-chat 2.5) demonstrated high accuracy in inclusion/exclusion decisions (99.38%, 98.94%, and 98.85%, respectively). Recall rates were 100.00%, 99.13%, and 98.26%, with statistically significant performance differences ( χ 2 = 5.99, P = 0.048), where GPT outperformed the other models. Exclusion reason concordance rates were 98.85%, 94.79%, and 96.47% ( χ 2 = 30.22, P < 0.001). In full-text screening, all models maintained perfect recall (100.00%), with accuracies of 100.00% (GPT), 100.00% (Kimi), and 99.45% (DeepSeek). Processing times averaged 1 to 5 s per article for title–abstract screening and 60 s for full-text processing (including PDF preprocessing). Conclusions: LitAutoScreener offers a new approach for efficient literature screening in drug intervention studies, achieving high accuracy and significantly improving screening efficiency.
261	title:thesocialscienceoflargelanguagemodelsasystematicreviewandcomputationaltaxonomy	2025	The Social Science of Large Language Models: A Systematic Review and Computational Taxonomy	Xiao Jia	OSF Preprints (OSF Preprints)	0	L1-openalex-llm-for-lit-search	1	SCREEN	exclude [E6] uses LLMs as a research subject, not as a review tool	This project preregisters a systematic review protocol on the social science of large language models (LLMs). LLMs such as GPT, Claude, and LLaMA are increasingly used in contexts that intersect with human social behavior, yet existing studies are scattered across disciplines. We will systematically search Web of Science, Scopus, ACM Digital Library, IEEE Xplore, PubMed, arXiv, Consensus, and Semantic Scholar, supplemented with hand-searching of top journals and conferences. Inclusion criteria focus on studies that directly involve LLMs and social science constructs; exclusion criteria filter out purely engineering applications and studies where LLMs are only used as tools. Screening will follow PRISMA 2020 guidelines, with two reviewers conducting independent title/abstract and full-text reviews. The expected outcome is a comprehensive overview of ~150 core studies, organized into a conceptual framework that highlights LLMs’ capacities and limitations. To ensure systematic categorization and to handle the large body of literature, we will employ an unsupervised machine learning approach to generate a computational taxonomy of the studies. This will involve clustering studies based on content similarity using sentence embeddings and dimensionality reduction techniques like UMAP. The resulting taxonomy will reveal the latent structure of the field and identify key research themes across various social science domains. This review will serve as a resource for researchers in AI, psychology, sociology, political science, and economics, and will provide methodological guidance for future work. Data extraction templates and coding sheets will be shared openly to enhance transparency and reproducibility.
262	doi:10.2196/84120	2025	Knowledge-Practice Performance Gap in Clinical Large Language Models: Systematic Review of 39 Benchmarks	Eun Jeong Gong	Journal of Medical Internet Research	41	L1-openalex-llm-for-lit-search	1	SCREEN	exclude [E6] uses LLMs as a research subject, not as a review tool	BACKGROUND: The evaluation of large language models (LLMs) in medicine has undergone a shift from knowledge-based testing to practice-based assessment, representing an evolution in how we measure artificial intelligence readiness for clinical deployment. While LLMs now routinely exceed human performance on medical licensing examinations, their translation to clinical practice remains poorly characterized. OBJECTIVE: This systematic review aims to categorize and analyze medical LLM benchmarks, examining performance patterns across different evaluation paradigms and identifying gaps in current assessment methodologies. METHODS: The protocol was registered at PROSPERO (CRD420251139729). Four databases (MEDLINE/PubMed, Embase/Ovid, Cochrane Library, and arXiv) were searched from inception to August 31, 2025, using keywords related to clinical medicine benchmarks in LLMs. Studies were included if they (1) investigated clinical medicine benchmarks in LLMs, (2) were published in English, and (3) were available in full-text. Studies were excluded if they evaluated nonmedical domains or lacked benchmark validation. Methodological quality was assessed using the Mixed Methods Appraisal Tool (version 2018) by 2 independent reviewers (κ=0.91). Due to heterogeneity in evaluation metrics preventing meta-analysis, narrative synthesis was conducted using structured categorization of benchmark types. RESULTS: From 3917 screened records, 39 medical LLM benchmarks were identified and categorized into 21 (54%) knowledge-based, 15 (38%) practice-based, and 3 (8%) hybrid frameworks. These benchmarks collectively encompass over 2.3 million questions across 45 languages and 172 medical specialties. Traditional knowledge-based benchmarks show saturation with leading models achieving 84%-90% accuracy on USMLE (United States Medical Licensing Examination)-style examinations, approaching or exceeding average physician performance. However, practice-based assessments reveal performance challenges, with specific benchmarks showing varied results: DiagnosisArena 45.82% (95% CI 42.9%-48.8%), MedAgentBench 69.67% (95% CI 64.2%-74.6%), and HealthBench 60% (95% CI 58.6%-61.3%) success rates, with practice-based benchmarks showing lower performance (45%-69%) compared to knowledge benchmarks (84%-90%). Task-specific analysis revealed differential performance patterns: factual retrieval maintained 85%-93% accuracy, clinical reasoning dropped to 50%-60%, diagnostic tasks achieved 45%-55% success, and safety assessment showed significant gaps at 40%-50% accuracy despite being life-critical. Geographic representation spans 6 continents with 18 (46%) benchmarks, incorporating non-English content. Quality assessment revealed 26% (10/39) of benchmarks had insufficient methodological reporting for complete evaluation. CONCLUSIONS: This systematic review provides the first comprehensive analysis quantifying the significant "knowledge-practice gap" in medical artificial intelligence: high performance on knowledge-based examinations (84%-90%) does not translate to clinical competence (45%-69%), with safety assessments at 40%-50%. Our findings provide quantitative evidence for regulators and health systems that examination scores are insufficient and misleading proxies for clinical readiness. This review concludes that autonomous deployment is not currently justifiable and that all evidence-based implementation strategies must mandate practice-oriented validation and robust human-in-the-loop oversight to ensure patient safety. TRIAL REGISTRATION: PROSPERO CRD420251139729; https://www.crd.york.ac.uk/PROSPERO/view/CRD420251139729.
263	doi:10.1101/2025.06.10.25329350	2025	Large Language Models for Sentiment Analysis in Healthcare: A Systematic Review Protocol	Ravi Shankar	medRxiv	0	L1-openalex-llm-for-lit-search	1	SCREEN	exclude [E6] uses LLMs as a research subject, not as a review tool	Abstract Large language models (LLMs) have emerged as powerful tools for sentiment analysis in healthcare, offering potential advantages in capturing contextual information and semantic relationships in complex medical text. Healthcare sentiment analysis presents unique challenges due to domain-specific terminology, privacy regulations, and the nuanced nature of patient experiences. This systematic review protocol outlines a comprehensive methodology to investigate the application of LLMs for sentiment analysis across healthcare settings, including analysis of patient feedback, social media content, and electronic health records. By synthesizing current evidence, we aim to provide insights for researchers, clinicians, and policymakers on the effectiveness, limitations, and ethical considerations of these advanced natural language processing techniques. We will conduct a systematic review following PRISMA-P 2015 guidelines and using the PICOS framework. The search strategy will encompass eight major databases (PubMed, Web of Science, Embase, CINAHL, MEDLINE, The Cochrane Library, PsycINFO, and Scopus) using a comprehensive search string combining terms related to LLMs, sentiment analysis, and healthcare contexts. We will include peer-reviewed studies published between 2018 (corresponding to BERT’s introduction) and March 2025 that focus on LLM applications for healthcare sentiment analysis with reported performance metrics or qualitative evaluations. Two independent reviewers will screen titles/abstracts and full texts, with disagreements resolved through discussion or third-reviewer consultation. Data extraction will capture study characteristics, research objectives, dataset details, LLM architecture specifications, fine-tuning approaches, performance metrics, and implementation challenges. Quality assessment will employ a modified QUADAS-2 tool and the Cochrane Risk of Bias tool. We will conduct narrative synthesis of the findings, organizing them thematically according to our research questions, with meta-analysis performed if study heterogeneity permits. PROSPERO registration number CRD420251012298 Strengths and limitations of this study This is the first systematic review to comprehensively examine large language models for sentiment analysis specifically within healthcare contexts, addressing a significant gap in the literature. The review’s rigorous methodology follows PRISMA-P guidelines and employs dual independent screening, data extraction, and quality assessment to ensure thoroughness and minimize bias. The inclusion of diverse healthcare text sources (patient feedback, social media, electronic health records) allows for a comprehensive understanding of LLM applications across the healthcare information ecosystem. By focusing on studies published since 2018 (when BERT was introduced), the review captures the most relevant technological developments while excluding outdated approaches. A limitation of this study is the expected heterogeneity across included studies (varying LLM architectures, datasets, metrics, and implementation contexts), which may preclude meaningful meta-analysis and limit definitive conclusions about relative performance, resulting in more descriptive than prescriptive findings.
264	doi:10.1016/j.cont.2025.102246	2025	322 - Role of Large Language Models in Urology: A systematic review and meta-analysis	Mohamed Javid Raja Iyub	Continence	1	L1-openalex-llm-for-lit-search	1	SCREEN	exclude [E6] uses LLMs as a research subject, not as a review tool	Hypothesis / aims of study Large Language Models (LLMs) have demonstrated transformative potential, with promising insights that could potentially enhance clinical practice. Their application in overall healthcare practices is increasingly noted, but their cumulative usage in the field of urology is less explored. Our study aimed to systematically review the current evidence on the usage of LLMs in urology and perform a quantitative analysis of comparable studies. Study design, materials and methods We searched electronic databases namely, Pubmed, Embase, Web of Science, and Scopus to include studies involving urological data or patients in which LLMs were utilized in clinical management or other relevant applications. The studies were grouped based on the application of the LLM that was being studied as: 1) Answering FAQs, 2) Patient Information Materials, 3) Clinical Practice, 4) Medical examination and 5) Other applications. The quality assessment was done utilising the Newcastle-Ottawa Scale. Results A total of 814 articles were found, and after the removal of duplicates and screening of the articles, 39 studies were included in our review. Various applications of LLMs in different domains were listed, and details regarding the training, outcomes, and limitations were studied. ChatGPT was the most commonly studied LLM. The pooled accuracy of the output of various LLMs on various medical examinations was found to be 63.24 (CI 53.69 – 72.72), and a forest plot was constructed. Studies analysing the output of LLMs in answering frequently asked questions reported an accuracy of 67.08% to 100%, with ChatGPT – 4 outperforming ChatGPT – 3.5. Interpretation of results The findings of this systematic review and meta-analysis underscore the growing role of LLMs in urology, highlighting their potential to enhance clinical practice across diverse applications. The pooled accuracy of 63.24% for LLM performance on medical examinations reflects moderate reliability, with variability likely influenced by differences in training data, model architecture, and evaluation methods. Notably, ChatGPT emerged as the most frequently studied LLM, with its latest version (ChatGPT-4) demonstrating superior accuracy (67.08% to 100%) in answering frequently asked questions compared to its predecessor, ChatGPT-3.5. This suggests that advancements in LLM iterations can significantly improve performance, particularly in patient-facing tasks. However, the results also reveal a lack of standardization in evaluation metrics and methodologies across studies, which poses challenges for consistent benchmarking and integration into clinical workflows. Concluding message LLMs' applications are diverse and have augmented urological practice. However, uniform evaluation methods and performance metrics for assessing the output generated for various purposes are needed to further streamline the use and synchronous incorporation of LLM in regular urological practice. Download: Download high-res image (104KB) Download: Download full-size image Figure 1 . Funding None Clinical Trial No Subjects None
265	doi:10.7759/cureus.94300	2025	Performance of ChatGPT and Large Language Models on Medical Licensing Exams Worldwide: A Systematic Review and Network Meta-Analysis With Meta-Regression	Alousious Kasagga	Cureus	10	L1-openalex-llm-for-lit-search	1	SCREEN	exclude [E6] uses LLMs as a research subject, not as a review tool	Large language models (LLMs) are increasingly being tested on national medical licensing examinations, yet existing research is fragmented across models, exam systems, and languages. This study is the first meta-analysis to systematically assess LLM performance across multiple medical licensing exams and languages using pooled estimates, network meta-analysis, and moderator-aware meta-regression. We synthesized accuracy data from 120 evaluations covering 10 exam systems in nine languages, identified through comprehensive searches of PubMed, Web of Science, and Institute of Electrical and Electronics Engineers (IEEE) Xplore, covering the period from 2021 to June 2025. The random-effects meta-analysis showed that 13 of 16 models exceeded the 60% passing threshold, with GPT-o1 leading at 95.4%, followed by DeepSeek-R1 (92.0%) and GPT-4o (89.4%). P-score rankings and network meta-analysis confirmed the superior performance of GPT-o1, while GPT-3.5 and LLaMA-13B consistently underperformed. Meta-regression revealed a significant variation in accuracy by model version, exam system, and language, with lower performance in Chinese and Japanese exams and higher performance in German and Peruvian settings. After adjustment for exam and language, GPT-o1 and DeepSeek-R1 achieved similar accuracy, both significantly higher than GPT-4. Model type, exam system, and language explained most of the between-study heterogeneity (R² =88.99%), and sensitivity analyses supported the robustness of the pooled estimates. Overall, several LLMs now approach or exceed the accuracy required to pass standardized medical licensing exams, supporting their potential role in medical education and decision support.
266	doi:10.1145/3769676	2025	Large Language Models for Cyber Security: A Systematic Literature Review	Hanxiang Xu	ACM Transactions on Software Engineering and Methodology	185	L2-openalex-llm-slr-software-eng	1	SCREEN	exclude [E6] uses LLMs as a research subject, not as a review tool	The rapid advancement of Large Language Models (LLMs) has opened up new opportunities for leveraging artificial intelligence in a variety of application domains, including cybersecurity. As the volume and sophistication of cyber threats continue to grow, there is an increasing need for intelligent systems that can automatically detect vulnerabilities, analyze malware, and respond to attacks. In this survey, we conduct a comprehensive review of the literature on the application of LLMs in cybersecurity (LLM4Security). By comprehensively collecting over 40K relevant papers and systematically analyzing 185 papers from top security and software engineering venues, we aim to provide a holistic view of how LLMs are being used to solve diverse problems across the cybersecurity domain. Through our analysis, we identify several key findings. First, we observe that LLMs are being applied to an expanding range of cybersecurity tasks, including vulnerability detection, malware analysis, and network intrusion detection. Second, we analyze application trends of different LLM architectures (such as encoder-only, encoder-decoder, and decoder-only) across security domains. Third, we identify increasingly sophisticated techniques for adapting LLMs to cybersecurity, such as advanced fine-tuning, prompt engineering, and external augmentation strategies. A significant emerging trend is the use of LLM-based autonomous agents, which represent a paradigm shift from single-task execution to orchestrating complex, multi-step security workflows. Furthermore, we find that the datasets used for training and evaluating LLMs are often limited, highlighting the need for more comprehensive datasets and the use of LLMs for data augmentation. Finally, we discuss the main challenges and opportunities for future research, including the need for more interpretable models, addressing the inherent security risks of LLMs, and their potential for proactive defense. Overall, our survey provides a comprehensive overview of the current state-of-the-art in LLM4Security and identifies several promising directions for future research. We believe that the insights and findings presented in this survey will contribute to the growing body of knowledge on the application of LLMs in cybersecurity and provide valuable guidance for researchers and practitioners working in this field.
267	arxiv:2509.11446	2025	Large Language Models (LLMs) for Requirements Engineering (RE): A Systematic Literature Review	Mohammad Amin Zadenoori	arXiv (Cornell University)	9	L2-openalex-llm-slr-software-eng	1	SCREEN	exclude [E6] uses LLMs as a research subject, not as a review tool	Large Language Models (LLMs) are finding applications in numerous domains, and Requirements Engineering (RE) is increasingly benefiting from their capabilities to assist with complex, language-intensive tasks. This paper presents a systematic literature review of 74 primary studies published between 2023 and 2024, examining how LLMs are being applied in RE. The study categorizes the literature according to several dimensions, including publication trends, RE activities, prompting strategies, and evaluation methods. Our findings indicate notable patterns, among which we observe substantial differences compared to previous works leveraging standard Natural Language Processing (NLP) techniques. Most of the studies focus on using LLMs for requirements elicitation and validation, rather than defect detection and classification, which were dominant in the past. Researchers have also broadened their focus and addressed novel tasks, e.g., test generation, exploring the integration of RE with other software engineering (SE) disciplines. Although requirements specifications remain the primary focus, other artifacts are increasingly considered, including issues from issue tracking systems, regulations, and technical manuals. The studies mostly rely on GPT-based models, and often use Zero-shot or Few-shot prompting. They are usually evaluated in controlled environments, with limited use in industry settings and limited integration in complex workflows. Our study outlines important future directions, such as leveraging the potential to expand the influence of RE in SE, exploring less-studied tasks, improving prompting methods, and testing in real-world environments. Our contribution also helps researchers and practitioners use LLMs more effectively in RE, by providing a list of identified tools leveraging LLMs for RE, as well as datasets.
268	doi:10.21203/rs.3.rs-5589929/v1	2025	Large Language Model for Requirements Engineering: A Systematic Literature Review	Javed Ali Khan	Research Square	14	L2-openalex-llm-slr-software-eng	1	SCREEN	exclude [E6] uses LLMs as a research subject, not as a review tool
269	arxiv:2506.15227	2025	Large Language Models for Unit Testing: A Systematic Literature Review	Quanjun Zhang	arXiv (Cornell University)	2	L2-openalex-llm-slr-software-eng	1	SCREEN	exclude [E6] uses LLMs as a research subject, not as a review tool	Unit testing is a fundamental practice in modern software engineering, with the aim of ensuring the correctness, maintainability, and reliability of individual software components. Very recently, with the advances in Large Language Models (LLMs), a rapidly growing body of research has leveraged LLMs to automate various unit testing tasks, demonstrating remarkable performance and significantly reducing manual effort. However, due to ongoing explorations in the LLM-based unit testing field, it is challenging for researchers to understand existing achievements, open challenges, and future opportunities. This paper presents the first systematic literature review on the application of LLMs in unit testing until March 2025. We analyze \numpaper{} relevant papers from the perspectives of both unit testing and LLMs. We first categorize existing unit testing tasks that benefit from LLMs, e.g., test generation and oracle generation. We then discuss several critical aspects of integrating LLMs into unit testing research, including model usage, adaptation strategies, and hybrid approaches. We further summarize key challenges that remain unresolved and outline promising directions to guide future research in this area. Overall, our paper provides a systematic overview of the research landscape to the unit testing community, helping researchers gain a comprehensive understanding of achievements and promote future research. Our artifacts are publicly available at the GitHub repository: https://github.com/iSEngLab/AwesomeLLM4UT.
270	doi:10.1109/fllm67465.2025.11390990	2025	Towards the Integration of Large Language Models into the Software Development Life Cycle: A Systematic Literature Review	Mohammadamin Madani	International Conference on Foundation and Large Language Models (FLLM)	1	L2-openalex-llm-slr-software-eng	1	SCREEN	exclude [E6] uses LLMs as a research subject, not as a review tool	Models (LLMs) are reshaping the Software Development Life Cycle (SDLC) by introducing new levels of automation and intelligent assistance across core phases, including requirements engineering, design, testing, and maintenance. Previous works have examined the adoption of LLM in these areas, but have overlooked its influence on traditional practices in SDLC. Therefore, this paper builds upon this foundation and investigates, from a phase-level perspective, how LLMs interact with the SDLC as a whole and whether their presence necessitates rethinking of conventional methodologies. To achieve this objective, this article reviews the state of the field using a systematic literature review method. Peer-reviewed studies published between 2019 and 2025, examining the integration, benefits, and limitations of LLM within SDLC are analyzed. The findings indicate that LLMs are most commonly applied in implementation and testing, where they improve productivity through code generation, test automation, and fault localisation. However, their adoption also introduces challenges related to model hallucination, contextual misalignment, computational overhead, and ethical risks such as bias and opacity. Therefore, we synthesise several LLM integration patterns and stress various risk mitigation strategies that can be adopted to enhance the integration of LLMs into software engineering practices. The study concludes by outlining practical implications for software teams and proposing future research directions to improve LLM’s integration with agile, DevOps and CI/CD workflows.
271	doi:10.1109/wsese66602.2025.00010	2025	On the Difficulties of Conducting and Replicating Systematic Literature Reviews Studies Using LLMs in Software Engineering	Katia Romero Felizardo	IEEE/ACM International Workshop on Methodological Issues with Empirical Studies in Software Engineering (WSESE)	6	L2-openalex-llm-slr-software-eng	1	SCREEN	include [LLM] difficulties replicating LLM-supported SLRs in SE	The Software Engineering (SE) community has adopted Systematic Literature Reviews (SLRs) to summarize the state-of-the-art in specific research topics. SLRs offer benefits such as synthesizing evidence from diverse studies to generate auditable results following a reproducible approach, and identifying research gaps for future exploration. However, the process is effort-intensive, prone to errors, and lays various challenges during their conduction. To overcome some of these issues, there is a growing belief that Large Language Models (LLMs) can support systematic literature reviews. While the literature has shown promising results in social sciences, more evidence of its accuracy is needed in technical fields like SE. In this context, studies and replications are essential in verifying the benefits and drawbacks of applying LLMs in systematic literature reviews. This paper discusses the difficulties in conducting and replicating studies that adopt LLMs to support systematic literature in SE. As an implication, we identified the challenges of adopting LLM in SLRs and offered a list of open issues for future research.
272	doi:10.5753/jserd.2025.5097	2025	Natural Language Processing in Software Engineering: A Systematic Literature Review	Gabriel Nogueira Pacheco	Journal of Software Engineering Research and Development	0	L2-openalex-llm-slr-software-eng	1	SCREEN	exclude [E6] uses LLMs as a research subject, not as a review tool	Context: Software engineering (SE) artifacts and documents, such as requirements specifications, user stories, test cases, and concepts of operations (ConOps), are typically written in natural language, making their manipulation challenging. Natural Language Processing (NLP) is a viable solution for managing these tasks. Objective: To conduct a systematic literature review to explore the current use of NLP in SE artifacts and tasks, supplemented by a tertiary study focusing on the emerging role of Large Language Models (LLMs) in software engineering research. Method: We searched digital libraries for relevant papers and applied inclusion and exclusion criteria to filter the primary studies. We then analyzed NLP techniques applied to SE documents and examined their usage in this context. Our research methodology followed Kitchenham and Charters' guidelines. Additionally, we conducted a tertiary study to synthesize findings from existing systematic literature reviews and surveys specifically addressing LLMs in software engineering. Results: We selected 60 primary studies to identify the most common methods for NLP pipelines, feature extraction, language models, and machine learning algorithms used in SE. We also assessed the purposes of these methods, their benefits for SE, their difficulty, and their contribution to SE advancement. The tertiary study revealed a rapid proliferation of LLM-focused research, with comprehensive reviews documenting exponential growth in publications and widespread adoption across diverse SE tasks. Conclusion: Requirements are the most frequently addressed artifacts using NLP techniques, with preprocessing and part-of-speech (POS) tagging being widely used. There is a notable increase in the use of large language models for various SE tasks, such as requirements elicitation, source code generation, bug fixing, and software testing. The tertiary study confirms that LLMs represent a pivotal shift in the research landscape, warranting dedicated investigation to understand their transformative impact on NLP applications in software engineering.
273	doi:10.1016/j.jss.2025.112758	2025	A systematic literature review of software engineering research on Jupyter notebook	Md. Saeed Siddik	Journal of Systems and Software	3	L2-openalex-llm-slr-software-eng	1	SCREEN	exclude [E6] uses LLMs as a research subject, not as a review tool	• This research provides the first comprehensive systematic literature review on software engineering research specifically targeting Jupyter notebooks, identifying 199 primary studies published up to September 2025 and categorizing them into 11 core software engineering topics. • This research reveals that a large portion of the studies have been published outside traditional software engineering venues, with Human-Computer Interaction conferences like ACM Conference on Human Factors in Computing Systems (CHI) being the top publishing venues, highlighting the interdisciplinary nature of Jupyter Notebook research. • This research identifies a reusability gap in existing research, showing that only 82 out of 199 studies offer usable replication packages, and most are hosted on GitHub instead of permanent repositories, which violates open science best practices. • This research identifies that notebook-specific solutions for software engineering issues such as testing, refactoring, and documentation are relatively underexplored. Future directions include resolving duplicated execution numbers, refactoring inter-notebook clones, and generating grouped documentation for coherent-code cells are future directions derived from our study. • This research proposes the integration of modern AI-based solutions into Jupyter notebooks to support various software engineering topics, including code search and code generation. Additionally, future research should leverage advanced AI techniques (e.g., large language models), to improve conversational AI-powered assistants for automated code generation by multi-step workflow automation in data science notebooks. • Although the paper exceeds the recommended length due to the inclusion of detailed tables, figures, and categorized analyses (covering 11 topics and 21 subtopics), we believe that this extended content is essential for clearly and completely reporting our findings. As the first systematic literature review in this domain, we have carefully structured the paper to ensure readability. We believe the length is justified by the value and breadth of this paper’s contributions. Context : Jupyter Notebook has emerged as a versatile tool that transforms how researchers, developers, and data scientists conduct and communicate their work. As the adoption of Jupyter notebooks continues to rise, so does the interest from the software engineering research community in improving the software engineering practices for Jupyter notebooks. Objective : The purpose of this study is to analyze trends, gaps, and methodologies used in software engineering research on Jupyter notebooks. Method : We selected 199 relevant publications up to September 2025, following established systematic literature review guidelines. We explored publication trends, categorized them based on software engineering topics, and reported findings based on those topics. Results : The most popular venues for publishing software engineering research on Jupyter notebooks are related to human-computer interaction instead of traditional software engineering venues. Researchers have addressed a wide range of software engineering topics on notebooks, such as code reuse, readability, and execution environment. Although reusability is one of the research topics for Jupyter notebooks, only 82 of the 199 studies can be reused based on their provided URLs. Additionally, most replication packages are not hosted on permanent repositories for long-term availability and adherence to open science principles. Conclusion : Solutions specific to notebooks for software engineering issues, including testing, refactoring, and documentation, are underexplored. Future research opportunities exist in automatic testing frameworks, refactoring clones between notebooks, and generating group documentation for coherent code cells.
274	arxiv:2505.16697	2025	Software Architecture Meets LLMs: A Systematic Literature Review	Larissa Schmid	arXiv (Cornell University)	4	L2-openalex-llm-slr-software-eng	1	SCREEN	exclude [E6] uses LLMs as a research subject, not as a review tool	Large Language Models (LLMs) are used for many different software engineering tasks. In software architecture, they have been applied to tasks such as classification of design decisions, detection of design patterns, and generation of software architecture design from requirements. However, there is little overview on how well they work, what challenges exist, and what open problems remain. In this paper, we present a systematic literature review on the use of LLMs in software architecture. We analyze 18 research articles to answer five research questions, such as which software architecture tasks LLMs are used for, how much automation they provide, which models and techniques are used, and how these approaches are evaluated. Our findings show that while LLMs are increasingly applied to a variety of software architecture tasks and often outperform baselines, some areas, such as generating source code from architectural design, cloud-native computing and architecture, and checking conformance remain underexplored. Although current approaches mostly use simple prompting techniques, we identify a growing research interest in refining LLM-based approaches by integrating advanced techniques.
275	doi:10.3390/ai6090206	2025	Unlocking the Potential of the Prompt Engineering Paradigm in Software Engineering: A Systematic Literature Review	Irdina Wanda Syahputri	AI	8	L2-openalex-llm-slr-software-eng	1	SCREEN	exclude [E6] uses LLMs as a research subject, not as a review tool	Prompt engineering (PE) has emerged as a transformative paradigm in software engineering (SE), leveraging large language models (LLMs) to support a wide range of SE tasks, including code generation, bug detection, and software traceability. This study conducts a systematic literature review (SLR) combined with a co-citation network analysis of 42 peer-reviewed journal articles to map key research themes, commonly applied PE methods, and evaluation metrics in the SE domain. The results reveal four prominent research clusters: manual prompt crafting, retrieval-augmented generation, chain-of-thought prompting, and automated prompt tuning. These approaches demonstrate notable progress, often matching or surpassing traditional fine-tuning methods in terms of adaptability and computational efficiency. Interdisciplinary collaboration among experts in AI, machine learning, and software engineering is identified as a key driver of innovation. However, several research gaps remain, including the absence of standardized evaluation protocols, sensitivity to prompt brittleness, and challenges in scalability across diverse SE applications. To address these issues, a modular prompt engineering framework is proposed, integrating human-in-the-loop design, automated prompt optimization, and version control mechanisms. Additionally, a conceptual pipeline is introduced to support domain adaptation and cross-domain generalization. Finally, a strategic research roadmap is presented, emphasizing future work on interpretability, fairness, and collaborative development platforms. This study offers a comprehensive foundation and practical insights to advance prompt engineering research tailored to the complex and evolving needs of software engineering.
276	doi:10.1145/3712003	2025	LLM-Based Multi-Agent Systems for Software Engineering: Literature Review, Vision, and the Road Ahead	Junda He	ACM Transactions on Software Engineering and Methodology	246	L2-openalex-llm-slr-software-eng	1	SCREEN	exclude [E6] uses LLMs as a research subject, not as a review tool	Integrating Large Language Models (LLMs) into autonomous agents marks a significant shift in the research landscape by offering cognitive abilities that are competitive with human planning and reasoning. This article explores the transformative potential of integrating Large Language Models into Multi-Agent (LMA) systems for addressing complex challenges in software engineering (SE). By leveraging the collaborative and specialized abilities of multiple agents, LMA systems enable autonomous problem-solving, improve robustness, and provide scalable solutions for managing the complexity of real-world software projects. In this article, we conduct a systematic review of recent primary studies to map the current landscape of LMA applications across various stages of the software development lifecycle (SDLC). To illustrate current capabilities and limitations, we perform two case studies to demonstrate the effectiveness of state-of-the-art LMA frameworks. Additionally, we identify critical research gaps and propose a comprehensive research agenda focused on enhancing individual agent capabilities and optimizing agent synergy. Our work outlines a forward-looking vision for developing fully autonomous, scalable, and trustworthy LMA systems, laying the foundation for the evolution of Software Engineering 2.0.
277	doi:10.5753/ise.2025.14892	2025	Investigating the use of Large Language Models in software security requirements: Results of a literature review	Amália Melo		1	L2-openalex-llm-slr-software-eng	1	SCREEN	exclude [E6] uses LLMs as a research subject, not as a review tool	This study investigates the application of Large Language Models (LLMs) in the context of security requirements engineering through the conduction of a rapid literature review. The review enabled the characterization of current research in this domain with respect to: (i) the purposes for which LLMs are employed in security requirements activities; (ii) the families of LLMs explored (e.g., GPT, BERT, LLaMA), their capabilities (e.g., classification, generation), and underlying architectures (e.g., encoder, decoder, encoder-decoder); (iii) the techniques adopted for conditioning or guiding LLM behavior; (iv) the datasets used to train, fine-tune, or validate these models; and (v) the evaluation metrics applied to assess the performance of LLMs in supporting security requirements tasks. The findings contribute to a structured understanding of the current state of research and highlight key trends, gaps, and opportunities for advancing the use of LLMs in secure software engineering.
278	doi:10.1016/j.infsof.2025.107960	2025	Using LLMs to enhance code quality: A systematic literature review	Nawaf Alomari	Information and Software Technology	4	L2-openalex-llm-slr-software-eng	1	SCREEN	exclude [E6] uses LLMs as a research subject, not as a review tool
279	doi:10.1109/access.2025.3535782	2025	Time-Series Large Language Models: A Systematic Review of State-of-the-Art	Shamsu Abdullahi	IEEE Access	41	L2-openalex-llm-slr-software-eng	1	SCREEN	exclude [E6] uses LLMs as a research subject, not as a review tool	Large Language Models (LLMs) have transformed Natural Language Processing (NLP) and Software Engineering by fostering innovation, streamlining processes, and enabling data-driven decision-making. Recently, the adoption of LLMs in time-series analysis has catalyzed the emergence of time-series LLMs, a rapidly evolving research area. Existing reviews provide foundational insights into time-series LLMs but lack a comprehensive examination of recent advancements and do not adequately address critical challenges in this domain. This Systematic Literature Review (SLR) bridges these gaps by analysing state-of-the-art contributions in time-series LLMs, focusing on architectural innovations, tokenisation strategies, tasks, datasets, evaluation metrics, and unresolved challenges. Using a rigorous methodology based on PRISMA guidelines, over 700 studies from 2020 to 2024 were reviewed, with 59 relevant studies selected from journals, conferences, and workshops. Key findings reveal advancements in architectures and novel tokenization strategies tailored for temporal data. Forecasting dominates the identified tasks with 79.66% of the selected studies, while classification and anomaly detection remain underexplored. Furthermore, the analysis reveals a strong reliance on datasets from the energy and transportation domains, highlighting the need for more diverse datasets. Despite these advancements, significant challenges persist, including tokenization inefficiencies, prediction hallucinations, and difficulties in modelling long-term dependencies. These issues hinder the robustness, scalability, and adaptability of time-series LLMs across diverse applications. To address these challenges, this SLR outlines a research roadmap emphasizing the improvement of tokenization methods, the development of mechanisms for capturing long-term dependencies, the mitigation of hallucination effects, and the design of scalable, interpretable models for diverse time-series tasks.
280	doi:10.3390/bdcc9120320	2025	A Systematic Literature Review of Retrieval-Augmented Generation: Techniques, Metrics, and Challenges	Andrew Brown	Big Data and Cognitive Computing	57	L2-openalex-llm-slr-software-eng	1	SCREEN	exclude [E6] uses LLMs as a research subject, not as a review tool	Background: Retrieval-augmented generation (RAG) aims to reduce hallucinations and outdated knowledge by grounding LLM outputs in retrieved evidence, but empirical results are scattered across tasks, systems, and metrics, limiting cumulative insight. Objective: We aimed to synthesise empirical evidence on RAG effectiveness versus parametric-only baselines, map datasets/architectures/evaluation practices, and surface limitations and research gaps. Methods: This systematic review was conducted and reported in accordance with PRISMA 2020. We searched the ACM Digital Library, IEEE Xplore, Scopus, ScienceDirect, and DBLP; all sources were last searched on 13 May 2025. This included studies from January 2020–May 2025 that addressed RAG or similar retrieval-supported systems producing text output, met citation thresholds (≥15 for 2025; ≥30 for 2024 or earlier), and offered original contributions; excluded non-English items, irrelevant works, duplicates, and records without accessible full text. Bias was appraised with a brief checklist; screening used one reviewer with an independent check and discussion. LLM suggestions were advisory only; 2025 citation thresholds were adjusted to limit citation-lag. We used a descriptive approach to synthesise the results, organising studies by themes aligned to RQ1–RQ4 and reporting summary counts/frequencies; no meta-analysis was undertaken due to heterogeneity of designs and metrics. Results: We included 128 studies spanning knowledge-intensive tasks (35/128; 27.3%), open-domain QA (20/128; 15.6%), software engineering (13/128; 10.2%), and medical domains (11/128; 8.6%). Methods have shifted from DPR + seq2seq baselines to modular, policy-driven RAG with hybrid/structure-aware retrieval, uncertainty-triggered loops, memory, and emerging multimodality. Evaluation remains overlap-heavy (EM/F1), with increasing use of retrieval diagnostics (e.g., Recall@k, MRR@k), human judgements, and LLM-as-judge protocols. Efficiency and security (poisoning, leakage, jailbreaks) are growing concerns. Discussion: Evidence supports a shift to modular, policy-driven RAG, combining hybrid/structure-aware retrieval, uncertainty-aware control, memory, and multimodality, to improve grounding and efficiency. To advance from prototypes to dependable systems, we recommend: (i) holistic benchmarks pairing quality with cost/latency and safety, (ii) budget-aware retrieval/tool-use policies, and (iii) provenance-aware pipelines that expose uncertainty and deliver traceable evidence. We note the evidence base may be affected by citation-lag from the inclusion thresholds and by English-only, five-library coverage. Funding: Advanced Research and Engineering Centre. Registration: Not registered.
281	doi:10.1145/3696673.3723053	2025	The Role of Generative AI Models in Requirements Engineering: A Systematic Literature Review	Poonkuzhali Vasudevan	ACM Southeast Conference (ACMSE)	9	L2-openalex-llm-slr-software-eng	1	SCREEN	exclude [E6] uses LLMs as a research subject, not as a review tool	The software engineering field is experiencing rapid growth, driven by recent advancements in Artificial Intelligence (AI), particularly in Generative AI (GenAI) and Large Language Models (LLMs). Requirements Engineering (RE), a critical phase in software development, involves gathering and defining software requirements. However, research on the impact of GenAI and LLMs within RE remains limited. This paper examines the adoption of GenAI in RE, with the aim of exploring its practical implications, identifying current research trends, and highlighting areas for future development. To achieve this, a systematic literature review was conducted, addressing three research questions and analyzing 44 studies published over the past decade. The findings reveal that GenAI models, especially LLMs, are extensively employed in a variety of RE tasks, underscoring the versatility and potential of LLMs in enhancing the RE process.
282	arxiv:2511.00776	2025	A Systematic Literature Review of Code Hallucinations in LLMs: Characterization, Mitigation Methods, Challenges, and Future Directions for Reliable AI	Cuiyun Gao	arXiv (Cornell University)	2	L2-openalex-llm-slr-software-eng	1	SCREEN	exclude [E6] uses LLMs as a research subject, not as a review tool	Model hallucination is one of the most critical challenges faced by Large Language Models (LLMs), especially in high-stakes code intelligence tasks. As LLMs become increasingly integrated into software engineering tasks, understanding and mitigating hallucination in code becomes essential. In this survey, we provide a systematic review of hallucination phenomena in code-oriented LLMs from four key perspectives. First, we begin by surveying 60 papers to define hallucination in the context of code and summarize its primary causes, such as data noise, exposure bias, and insufficient semantic grounding, while also tracing recent trends in literature across natural language processing (NLP) and software engineering communities. Second, we review model hallucination surveys in a broader span and summarize representative hallucination mitigation strategies, such as knowledge-enhanced generation, constrained decoding, and post-editing. Third, we review approaches targeted for code intelligence and highlight code-specific challenges that aggravate hallucination, including syntax sensitivity, strict type systems, and dependence on external libraries. Meanwhile, we analyze how emerging code intelligence tasks, e.g., program analysis, symbolic execution, and unit testing, are utilized to detect and mitigate hallucinations. Fourth, we summarize current evaluation benchmarks, ranging from static metrics to dynamic checks, e.g., compilation and execution correctness, and emphasize the need for hallucination-oriented benchmarks.
283	doi:10.1145/3766552	2025	Metamorphic Testing of Deep Code Models: A Systematic Literature Review	Ali Asgari	ACM Transactions on Software Engineering and Methodology	3	L2-openalex-llm-slr-software-eng	1	SCREEN	exclude [E6] uses LLMs as a research subject, not as a review tool	Large language models and deep learning models designed for code intelligence have revolutionized the software engineering field due to their ability to perform various code-related tasks. These models can process source code and software artifacts with high accuracy in tasks such as code completion, defect detection, and code summarization; therefore, they can potentially become an integral part of modern software engineering practices. Despite these capabilities, robustness remains a critical quality attribute for deep-code models as they may produce different results under varied and adversarial conditions (e.g., variable renaming). Metamorphic testing has become a widely used approach to evaluate models’ robustness by applying semantic-preserving transformations to input programs and analyzing the stability of model outputs. While prior research has explored testing deep learning models, this systematic literature review focuses specifically on metamorphic testing for deep code models. By studying 45 primary papers, we analyze the transformations, techniques, and evaluation methods used to assess robustness. Our review summarizes the current landscape, identifying frequently evaluated models, programming tasks, datasets, target languages, and evaluation metrics, and highlights key challenges and future directions for advancing the field.
284	doi:10.1016/j.infsof.2025.107979	2025	Quality assessment of software requirements using artificial intelligence methods: A systematic literature review	Elise Wolf	Information and Software Technology	7	L2-openalex-llm-slr-software-eng	1	SCREEN	exclude [E6] uses LLMs as a research subject, not as a review tool	Context: The quality of requirements specifications is a critical success factor in software development. Assuring high-quality requirements, specifically in an automated way, poses a significant challenge due to their unstructured and multi-modal character. With the rise of deep learning and large language models (LLMs), new opportunities have developed to assess the quality of requirements automatically, particularly user stories in the context of agile software engineering, where short development cycles require efficient tool support. Objective: This study aims to systematically review and investigate the current landscape of approaches based on artificial intelligence techniques such as natural language processing and deep learning for assessing the quality of software requirements. The investigation focuses on the artificial intelligence techniques adopted, quality aspects considered, datasets used to tune and evaluate the proposed approaches, and their performance. Method: We conducted a systematic literature review of 26 peer-reviewed papers published between 2019 and 2025. We selected the papers after a title and abstract review of 353 papers identified through a literature databases query and forward–backward snowballing. Results: The results reveal significant overlap among considered quality aspects, which can be mapped onto the higher-order requirements quality model INVEST. Most studies focus on assessing requirement quality rather than improving requirements and rely heavily on synthetic and public datasets. LLMs have rapidly gained popularity since 2023, though model evaluation strategies remain inconsistent. Metrics such as accuracy, precision, recall, and F1-Score are common, yet a few studies use semantic or expert-based evaluations. Conclusion: The field is evolving toward LLM-driven, semantically rich models, yet lacks methodological standardization, reproducible datasets for evaluating the models, and integration of the approaches with real-world requirements engineering processes. Future work should address these limitations by developing benchmark datasets, standardizing evaluation metrics, and exploring hybrid systems that combine AI-based and traditional requirements quality assurance approaches.
285	doi:10.1109/tse.2025.3586411	2025	Open Source, Hidden Costs: A Systematic Literature Review on OSS License Management	Boyuan Li	IEEE Transactions on Software Engineering	5	L2-openalex-llm-slr-software-eng	1	SCREEN	exclude [E6] uses LLMs as a research subject, not as a review tool	Integrating third-party software components is a common practice in modern software development, offering significant advantages in terms of efficiency and innovation. However, this practice is fraught with risks related to software licensing. A lack of understanding may lead to disputes, which can pose serious legal and operational challenges. To these ends, both academia and industry have conducted various investigations and proposed solutions and tools to deal with these challenges. However, significant limitations still remain. Moreover, the rapid evolution of open-source software (OSS) licenses, as well as the rapidly incorporated generative software engineering techniques, such as large language models for code (CodeLLMs), are placing greater demands on the systematic management of software license risks. To unveil the severe challenges and explore possible future directions, we conduct the first systematic literature review (SLR) on 80 carefully selected OSS license-related papers, classifying existing research into three key categories, i.e., license identification, license risk assessment, and license risk mitigation. Based on these, we discuss challenges in existing solutions, conclude the opportunities to shed light on future research directions and offer practical recommendations for practitioners.We hope this thorough review will help bridge the gaps between academia and industry and accelerate the ecosystem-wide governance of legitimate software risks within the software engineering community.
286	title:experienceswithintegratingaitoolsintosoftwareengineeringeducationaliteraturereview	2025	Experiences with integrating AI tools into software engineering education: A literature review	Ilari Matikainen	Työväentutkimus Vuosikirja	0	L2-openalex-llm-slr-software-eng	1	SCREEN	exclude [E6] uses LLMs as a research subject, not as a review tool	Due to landmark developments in large language model technologies during recent years, a large influx of novel artificial intelligence software has created newfound interest in utilizing AI tools for a variety of different use cases. This thesis looks into the early experiences of integrating AI tools into software engineering education, and examines the effects that their use has had on learning and teaching software engineering. The study is conducted through a literature review. The data used for the literature review is a collection of previous studies on the use of AI tools in the classroom, with focus on their practical applications. The main findings are that both ChatGPT and custom-developed educational AI tools are being utilized actively, and that their use has generally had a positive effect on multiple aspects of software engineering education. Negative effects were identified too, however. These findings suggest that artificial intelligence will most likely have a larger role in the future education of software engineers, and different ways of utilizing AI tools in educational interventions should be further researched.
287	doi:10.20533/ijicr.2042.4655.2025.0165	2025	Attention Mechanism Enhancements in Modern Machine Learning	B. Thomas	International Journal of Intelligent Computing Research	0	S1-openalex-qknorm	1	SCREEN	exclude [E5] title/abstract: not about attention-logit control or LM training stability	Attention mechanisms have become a cornerstone of modern machine learning, enabling models (especially Transformers) to weigh the importance of different input elements dynamically.Since the introduction of the Transformer and its Multi-Head Self-Attention (MHA) in 2017, attention-based models have achieved state-of-the-art results in NLP, computer vision, and other domains.However, the flexibility of attention comes at a high computational and memory cost: naïve self-attention requires O(n²) operations and memory with respect to sequence length n.This poses challenges for long sequences and efficient inference, motivating a surge of research into faster and more memory-efficient attention mechanisms.In this article, we review the fundamentals of attention and highlight recent advances aimed at improving attention speed and efficiency, including Multi-Query and Grouped-Query Attention, Rotary Positional Embeddings (RoPE), Query-Key Normalization (QK Norm), hybrid Transformer models, and high-performance implementations like FlashAttention.
288	doi:10.1109/iceict66683.2025.11159965	2025	Enhanced Monocular Depth Estimation Based on Improved Self-Attention Mechanisms and Composite Loss Functions	Mingshen Huang	IEEE International Conference on Electronic Information and Communication Technology (ICEICT)	0	S1-openalex-qknorm	1	SCREEN	exclude [E5] title/abstract: not about attention-logit control or LM training stability	Monocular depth estimation, as a key technology in computer vision, has been widely applied in autonomous driving, robotic navigation, and other fields. However, balancing the accuracy of global depth prediction with the fine recovery of local details remains challenging. This paper presents an improved monocular depth estimation framework based on the powerful Depth Anything V2 pre-trained model. First, we fine-tune the ViT-Large pre-trained weights and integrate Query-Key Normalization (QK-Norm), Relative Position Bias (RelPosBias), and Rotary Position Embedding (RoPE) mechanisms into its core self-attention module. These mechanisms enhance the model's ability to perceive spatial information and relative positional relationships, improving the stability and position sensitivity of feature representation. Second, to address common issues of local discontinuities and edge blurring in depth prediction, we construct a composite loss function consisting of improved Scale-invariant Logarithmic Loss (SiLogLoss), Gradient Consistency Loss (GradientLoss), and Edge Smoothness Loss (EdgeSmoothnessLoss), providing supervision from global error, depth map structure, and edge details. We conducted comprehensive experimental evaluations on the KITTI dataset. Results demonstrate that our method achieves significant improvements over the original Depth Anything V2 model in key performance metrics: reducing the Absolute Relative Error (AbsRel) from 0.074 to 0.064, and increasing the $\delta_1$ accuracy from 0.946 to 0.967, validating the effectiveness of our proposed strategies.
289	title:dualityweightdecayandmetrizeddeeplearning	2025	Duality, Weight Decay, and Metrized Deep Learning	Newhouse, Laker	DSpace@MIT (Massachusetts Institute of Technology)	0	S1-openalex-qknorm	1	SCREEN	exclude [E4] near-duplicate of 2507.13338	The Muon optimizer has shown convincing evidence that it is faster and more scalable than AdamW for deep learning training, setting speed records for training NanoGPT and scaling up to models with 16B parameters. The theory that led to Muon is called metrized deep learning, a method that suggests assigning norms to each part of a neural network. Chapter 1 begins with an accessible explanation of metrized deep learning, including one of its recurring tools: odd polynomial iterations that act directly on singular values. Chapter 2 reviews duality, a way to modify the gradient that seeks to decrease the loss the most while disturbing the model the least. Pedagogically, duality links four popular optimizers—SGD, Adam, Shampoo, and Muon—under a common framework, steepest descent under a norm. Practically, experiments suggest that duality-based optimizers train faster than AdamW and transfer learning rate across width. Chapter 3 develops tools to enforce weight norm constraints during training, conferring provable and upfront Lipschitz guarantees for transformers. We find that optimizer dynamics matter: switching from AdamW to Muon improves standard weight regularization methods—weight decay and spectral normalization—allowing models to reach equal performance with a lower Lipschitz bound. Leveraging that Muon’s update has a fixed spectral norm, we co-design a weight constraint method called spectral cap that improves the Lipschitz vs. performance tradeoff for MLPs and 2M parameter transformers. Our 4-Lipschitz transformer on Shakespeare text reaches validation accuracy 60%. Scaling to 145M parameters, our 600-Lipschitz transformer reaches 21% accuracy on internet text. However, to match the NanoGPT baseline validation accuracy of 39.4%, our Lipschitz upper bound increases to 10^274. Nonetheless, our Lipschitz transformers train without stability measures such as layer norm, QK norm, and tanh logit softcapping.
290	title:costeffectivedevelopmentofeducationalchatbotstrainingandinferenceoptimization	2025	Cost-Effective Development of Educational Chatbots: Training and Inference Optimization	Aibek Abilgazym	Nazarbayev University Repository (Nazarbayev University)	0	S1-openalex-qknorm	1	SCREEN	exclude [E5] title/abstract: not about attention-logit control or LM training stability	Large Language Models (LLMs) have revolutionized natural language processing tasks and show promise for interactive language learning applications. However, training these models and deploying them on resource-constrained devices (like smartphones or Raspberry Pi) is challenging due to their size and computational demands. This thesis presents an accelerated training pipeline for the GPT-2 124M model – the small- est GPT-2 – enabling it to be trained to competitive performance in under 3 minutes (using 0.8 billion tokens) on high-end hardware, a dramatic reduction compared to the original GPT-2 which required about 40 GB of text and multiple days of training. We achieve this speed-up through a combination of architectural optimizations (e.g. Rotary Positional Embeddings, ReLU² activations, QK-normalization, FlexAttention mechanism, and etc.), a custom Muon optimizer for faster convergence, and strate- gic hyperparameter tuning. The model was trained on the cleaned and deduplicated FineWeb dataset. The resulting model was then quantized to 8-bit, reducing memory by a factor of four with minimal loss in language modeling accuracy. We demonstrate the model’s effectiveness in an educational context, focusing on tasks like interac- tive conversation practice and grammar correction, and deployed on edge devices, including an 8GB Raspberry Pi. Benchmarks show that the optimized 124M model maintains language modelling performance (HellaSwag benchmark) while achieving real-time inference speeds within the limited memory ( 1 GB) e of these devices. This work highlights that with the right optimizations, even moderately sized LLMs can be trained and run off-line on personal devices for language learning, bridging the gap between large cloud-based models and accessible on-device AI tutors.
291	doi:10.48448/nht8-e622	2025	Variance Sensitivity Induces Attention Entropy Collapse and Instability in Transformers	Association for Computational Linguistics 2025	Underline Science Inc.	0	S2-openalex-logit-instability	1	SCREEN	include [IM] variance sensitivity -> attention entropy collapse	Attention-based language models typically rely on the softmax function to convert attention logits into probability distributions. However, this mechanism can lead to attention entropy collapse, where attention is focused on a single token, causing training instability. In this work, we identify the high variance sensitivity of softmax as a primary cause of this collapse. We show that entropy-stable attention mechanisms, which either control or are insensitive to the variance of attention logits, can prevent entropy collapse and enable more stable training. We provide empirical evidence of this effect in both large language models (LLMs) and a small Transformer model composed solely of self-attention and support our findings with theoretical analysis. Moreover, we identify that the concentration of attention probabilities increases the probability matrix norm, leading to a gradient exploding.
292	doi:10.1109/fuzz62266.2025.11152151	2025	Fuzzy Confidence Attention: Harnessing Token-Level Uncertainty for Robust Transformer Models	Taniya Seth	IEEE International Conference on Fuzzy Systems (FUZZ-IEEE)	0	S2-openalex-logit-instability	1	SCREEN	exclude [E5] title/abstract: not about attention-logit control or LM training stability	Transformer architectures have transformed natural language processing by capturing intricate dependencies through self-attention. Yet, conventional attention mechanisms lack a means to address token-level uncertainty, leaving models vulnerable to noisy, uncertain or unreliable inputs in real-world applications. In this paper, we propose fuzzy confidence attention—a novel mechanism that integrates token-level confidence estimates into the transformer’s attention framework via fuzzy sets. By mapping token confidence scores to fuzzy membership values using a differentiable, piecewise linear function, our method modulates attention logits to dynamically down-weight unreliable tokens. Experimental evaluations on controlled datasets reveal that our approach systematically reallocates attention, as shown by both qualitative heatmap visualizations and quantitative metrics such as Mean Noisy Token Attention and average KL divergence. These results demonstrate improved robustness and interpretability compared to standard attention mechanisms. FCA thus paves the way for more resilient transformer models in noisy and uncertain environments, with promising avenues for future work in large-scale and multimodal settings.
293	arxiv:2510.05491	2025	NorMuon: Making Muon more efficient and scalable	Zi-Chong Li	arXiv.org	88	snowball-back-imu1	1	SCREEN	exclude [E5] title/abstract: not about attention-logit control or LM training stability	The choice of optimizer significantly impacts the training efficiency and computational costs of large language models (LLMs). Recently, the Muon optimizer has demonstrated promising results by orthogonalizing parameter updates, improving optimization geometry through better conditioning. Despite Muon's emergence as a candidate successor to Adam, the potential for jointly leveraging their strengths has not been systematically explored. In this work, we bridge this gap by proposing NorMuon (Neuron-wise Normalized Muon), an optimizer that synergistically combines orthogonalization with neuron-level adaptive learning rates. Our analysis reveals that while Muon effectively reduces condition numbers, the resulting updates exhibit highly non-uniform neuron norms, causing certain neurons to dominate the optimization process. NorMuon addresses this imbalance by maintaining second-order momentum statistics for each neuron and applying row-wise normalization after orthogonalization, ensuring balanced parameter utilization while preserving Muon's conditioning benefits. To enable practical deployment at scale, we develop an efficient distributed implementation under the FSDP2 framework that strategically distributes orthogonalization computations across devices. Experiments across multiple model scales demonstrate that NorMuon consistently outperforms both Adam and Muon, achieving 21.74% better training efficiency than Adam and 11.31% improvement over Muon on 1.1 B pretraining setting, while maintaining a comparable memory footprint to Muon. Our findings suggest that orthogonalization and adaptive learning rates are complementary rather than competing approaches, opening new avenues for optimizer design in large-scale deep learning.
294	arxiv:2502.05795	2025	The Curse of Depth in Large Language Models	Wenfang Sun	Neural Information Processing Systems	70	snowball-back-imu1	1	SCREEN	include [IM] depth effects in LLMs (H3 context)	In this paper, we introduce the Curse of Depth, a concept that highlights, explains, and addresses the recent observation in modern Large Language Models (LLMs) where nearly half of the layers are less effective than expected. We first confirm the wide existence of this phenomenon across the most popular families of LLMs such as Llama, Mistral, DeepSeek, and Qwen. Our analysis, theoretically and empirically, identifies that the underlying reason for the ineffectiveness of deep layers in LLMs is the widespread usage of Pre-Layer Normalization (Pre-LN). While Pre-LN stabilizes the training of Transformer LLMs, its output variance exponentially grows with the model depth, which undesirably causes the derivative of the deep Transformer blocks to be an identity matrix, and therefore barely contributes to the training. To resolve this training pitfall, we propose LayerNorm Scaling (LNS), which scales the variance of output of the layer normalization inversely by the square root of its depth. This simple modification mitigates the output variance explosion of deeper Transformer layers, improving their contribution. Across a wide range of model sizes (130M to 7B), our experiments show that LNS consistently outperforms previous normalization and scaling techniques in enhancing LLM pre-training performance. Moreover, this improvement seamlessly carries over to supervised fine-tuning. All these gains can be attributed to the fact that LayerNorm Scaling enables deeper layers to contribute more effectively during training. Our code is available at \href{https://github.com/lmsdss/LayerNorm-Scaling}{LayerNorm-Scaling}.
295	arxiv:2510.23912	2025	Key and Value Weights Are Probably All You Need: On the Necessity of the Query, Key, Value weight Triplet in Self-Attention Transformers	Marko Karbevski		2	snowball-fwd-henry2020	1	SCREEN	exclude [E5] title/abstract: not about attention-logit control or LM training stability	We theoretically investigate whether the Query, Key, Value weight triplet can be reduced in encoder-only and decoder-only transformers. Under mild assumptions, we prove that one of the Query, Key or Value weights are redundant and can be replaced with the identity matrix, reducing attention parameters by 25\%. If applied to the Query or Key weights, this also simplifies optimization: attention logits depend on a single learned weight matrix rather than on a product of two. Validating the Query weight removal on decoder-only GPT-style small models trained from scratch, we find that reduced models match baseline performance despite fewer parameters, and outperform baselines when saved parameters are reallocated. Our analysis has also led us to a structural expressivity boundary: in the mathematically tractable ReLU setting, skip connections push MLPs into a generically disjoint function class at fixed width. These findings motivate investigation across modalities and at scale, where the observed stability and efficiency gains may prove most consequential.
296	arxiv:2510.22777	2025	SeeDNorm: Self-Rescaled Dynamic Normalization	Wenrui Cai	arXiv.org	7	snowball-fwd-henry2020	1	SCREEN	exclude [E5] title/abstract: not about attention-logit control or LM training stability	Normalization layer constitutes an essential component in neural networks. In transformers, the predominantly used RMSNorm constrains vectors to a unit hypersphere, followed by dimension-wise rescaling through a learnable scaling coefficient $\gamma$ to maintain the representational capacity of the model. However, RMSNorm discards the input norm information in forward pass and a static scaling factor $\gamma$ may be insufficient to accommodate the wide variability of input data and distributional shifts, thereby limiting further performance improvements, particularly in zero-shot scenarios that large language models routinely encounter. To address this limitation, we propose SeeDNorm, which enhances the representational capability of the model by dynamically adjusting the scaling coefficient based on the current input, thereby preserving the input norm information and enabling data-dependent, self-rescaled dynamic normalization. During backpropagation, SeeDNorm retains the ability of RMSNorm to dynamically adjust gradient according to the input norm. We provide a detailed analysis of the training optimization for SeedNorm and proposed corresponding solutions to address potential instability issues that may arise when applying SeeDNorm. We validate the effectiveness of SeeDNorm across models of varying sizes in large language model pre-training as well as supervised and unsupervised computer vision tasks. By introducing a minimal number of parameters and with neglligible impact on model efficiency, SeeDNorm achieves consistently superior performance compared to previously commonly used normalization layers such as RMSNorm and LayerNorm, as well as element-wise activation alternatives to normalization layers like DyT.
297	arxiv:2509.11865	2025	Tenma: Robust Cross-Embodiment Robot Manipulation with Diffusion Transformer	Travis Davies	arXiv.org	4	snowball-fwd-henry2020	1	SCREEN	exclude [E5] title/abstract: not about attention-logit control or LM training stability	Scaling Transformer policies and diffusion models has advanced robotic manipulation, yet combining these techniques in lightweight, cross-embodiment learning settings remains challenging. We study design choices that most affect stability and performance for diffusion-transformer policies trained on heterogeneous, multimodal robot data, and introduce Tenma, a lightweight diffusion-transformer for bi-manual arm control. Tenma integrates multiview RGB, proprioception, and language via a cross-embodiment normalizer that maps disparate state/action spaces into a shared latent space; a Joint State-Time encoder for temporally aligned observation learning with inference speed boosts; and a diffusion action decoder optimized for training stability and learning capacity. Across benchmarks and under matched compute, Tenma achieves an average success rate of 88.95% in-distribution and maintains strong performance under object and scene shifts, substantially exceeding baseline policies whose best in-distribution average is 18.12%. Despite using moderate data scale, Tenma delivers robust manipulation and generalization, indicating the great potential for multimodal and cross-embodiment learning strategies for further augmenting the capacity of transformer-based imitation learning policies.
298	arxiv:2505.15353	2025	Establishing a Scale for Kullback-Leibler Divergence in Language Models Across Various Settings	Ryōji Kishino		3	snowball-fwd-henry2020	1	SCREEN	exclude [E5] title/abstract: not about attention-logit control or LM training stability	Log-likelihood vectors define a common space for comparing language models as probability distributions, enabling unified comparisons across heterogeneous settings. We extend this framework to training checkpoints and intermediate layers, and establish a consistent scale for KL divergence across pretraining, model size, random seeds, quantization, fine-tuning, and layers. Analysis of Pythia pretraining trajectories further shows that changes in log-likelihood space, as measured by the scaling behavior of KL divergence, are much smaller than in weight space, resulting in subdiffusive learning trajectories and early stabilization of language-model behavior despite weight drift.
299	arxiv:2502.11646	2025	Hyper-SET: Designing Transformers via Hyperspherical Energy Minimization	Yun-Zhe Hu		2	snowball-fwd-henry2020	1	SCREEN	exclude [E5] title/abstract: not about attention-logit control or LM training stability	Transformer-based models have achieved remarkable success, but their core components, Transformer layers, are largely heuristics-driven and engineered from the bottom up, calling for a prototypical model with high interpretability and practical competence. To this end, we conceptualize a principled, top-down approach grounded in energy-based interpretation. Specifically, we formalize token dynamics as a joint maximum likelihood estimation on the hypersphere, featuring two properties: semantic alignment in the high-dimensional space and distributional uniformity in the low-dimensional space. By quantifying them with extended Hopfield energy functions, we instantiate this idea as a constrained energy minimization problem, which enables designs of symmetric attention and feedforward modules with RMS normalization. We further present \textit{Hyper-Spherical Energy Transformer} (Hyper-SET), a recurrent-depth alternative to vanilla Transformers naturally emerging from iterative energy optimization on the hypersphere. With shared parameters across layers, Hyper-SET can scale to arbitrary depth with fewer parameters. Theoretically grounded and compact, it achieves competitive or superior performance across diverse tasks, including Sudoku solving, image classification, and masked image modeling. We also design novel variations under the proposed general principle, such as linear attention and gated feedforward layer. Moreover, we showcase its scalability with depth-wise LoRA. Our results highlight Hyper-SET as a step toward interpretable and principled Transformer design.
300	arxiv:2512.16248	2025	Sigma-MoE-Tiny Technical Report	Qing-Guo Hu	arXiv.org	1	snowball-fwd-vit22b	1	SCREEN	exclude [E5] title/abstract: not about attention-logit control or LM training stability	Mixture-of-Experts (MoE) has emerged as a promising paradigm for foundation models due to its efficient and powerful scalability. In this work, we present Sigma-MoE-Tiny, an MoE language model that achieves the highest sparsity compared to existing open-source models. Sigma-MoE-Tiny employs fine-grained expert segmentation with up to 96 experts per layer, while activating only one expert for each token, resulting in 20B total parameters with just 0.5B activated. The major challenge introduced by such extreme sparsity lies in expert load balancing. We find that the widely-used load balancing loss tends to become ineffective in the lower layers under this setting. To address this issue, we propose a progressive sparsification schedule aiming to balance expert utilization and training stability. Sigma-MoE-Tiny is pre-trained on a diverse and high-quality corpus, followed by post-training to further unlock its capabilities. The entire training process remains remarkably stable, with no occurrence of irrecoverable loss spikes. Comprehensive evaluations reveal that, despite activating only 0.5B parameters, Sigma-MoE-Tiny achieves top-tier performance among counterparts of comparable or significantly larger scale. In addition, we provide an in-depth discussion of load balancing in highly sparse MoE models, offering insights for advancing sparsity in future MoE architectures. Project page: https://qghuxmu.github.io/Sigma-MoE-Tiny Code: https://github.com/microsoft/ltp-megatron-lm
301	arxiv:2512.01059	2025	Parameter Reduction Improves Vision Transformers: A Comparative Study of Sharing and Width Reduction	Anantha Padmanaban Krishna Kumar	arXiv.org	1	snowball-fwd-vit22b	1	SCREEN	exclude [E5] title/abstract: not about attention-logit control or LM training stability	Although scaling laws and many empirical results suggest that increasing the size of Vision Transformers often improves performance, model accuracy and training behavior are not always monotonically increasing with scale. Focusing on ViT-B/16 trained on ImageNet-1K, we study two simple parameter-reduction strategies applied to the MLP blocks, each removing 32.7\% of the baseline parameters. Our \emph{GroupedMLP} variant shares MLP weights between adjacent transformer blocks and achieves 81.47\% top-1 accuracy while maintaining the baseline computational cost. Our \emph{ShallowMLP} variant halves the MLP hidden dimension and reaches 81.25\% top-1 accuracy with a 38\% increase in inference throughput. Both models outperform the 86.6M-parameter baseline (81.05\%) and exhibit substantially improved training stability, reducing peak-to-final accuracy degradation from 0.47\% to the range 0.03\% to 0.06\%. These results suggest that, for ViT-B/16 on ImageNet-1K with a standard training recipe, the model operates in an overparameterized regime in which MLP capacity can be reduced without harming performance and can even slightly improve it. More broadly, our findings suggest that architectural constraints such as parameter sharing and reduced width may act as useful inductive biases, and highlight the importance of how parameters are allocated when designing Vision Transformers. All code is available at: https://github.com/AnanthaPadmanaban-KrishnaKumar/parameter-efficient-vit-mlps.
302	doi:10.1109/icicml67980.2025.11333455	2025	Research on Image Classification Based on Deep Learning	K. Ran	2025 4th International Conference on Image Processing, Computer Vision and Machine Learning (ICICML)	0	snowball-fwd-vit22b	1	SCREEN	exclude [E5] title/abstract: not about attention-logit control or LM training stability	This paper focuses on the study of feature extraction and fusion methods in deep learning models for image classification tasks. To address the limitations of traditional convolutional neural networks (CNNs) — such as gradient vanishing and parameter redundancy when modeling global features and training deep architectures — this study introduces the skip-connection mechanism of the Residual Network (ResNet) to enhance feature propagation and training stability. Building upon this foundation, we integrate the global self-attention modeling capability of the Vision Transformer (ViT) and propose a hybrid pyramid-structured image classification model that combines convolutional features, residual information, and attention mechanisms. In the shallow layers, the model performs local feature extraction and multi-scale fusion; in the middle layers, residual modules ensure continuous feature flow; and in the high layers, the self-attention mechanism captures global semantic relationships within images. Through this hierarchical design, the model achieves efficient cooperation between local and global representations. Experimental results demonstrate that the proposed model achieves superior recognition accuracy and strong generalization ability across multiple image classification benchmarks, providing an effective solution for feature representation and classification in complex visual scenarios.
303	arxiv:2506.21137	2025	Norm$\times$Direction: Restoring the Missing Query Norm in Vision Linear Attention	Wei-Kang Meng		4	snowball-fwd-vit22b	1	SCREEN	exclude [E5] title/abstract: not about attention-logit control or LM training stability	Linear attention mitigates the quadratic complexity of softmax attention but suffers from a critical loss of expressiveness. We identify two primary causes: (1) The normalization operation cancels the query norm, which breaks the correlation between a query's norm and the spikiness (entropy) of the attention distribution as in softmax attention. (2) Standard techniques for enforcing non-negativity cause destructive information loss by nullifying valid inner-product interactions. To address these challenges, we introduce NaLaFormer, a novel linear attention mechanism built upon a norm$\times$direction (ND) decomposition of the query and key vectors. We leverage each component to solve a distinct problem: The query norm is injected into our kernel to create a query-norm-aware map that restores the attention distribution's spikiness. The direction vectors are processed by a geometric, cosine-based similarity metric that guarantees non-negativity while preserving the rich, fine-grained information of the inner product. We validate NaLaFormer through a comprehensive multi-modal evaluation, where it sets new state-of-the-art benchmarks for linear attention. Our model achieves up to a 7.5% accuracy gain on ImageNet-1K and a 4.7% mIoU improvement on ADE20K over comparable baselines. It demonstrates profound efficiency, reducing peak memory by a transformative 92.3% in token-intensive super-resolution tasks (70K+ tokens). NaLaFormer's versatility is further confirmed as it surpasses strong baselines like Mamba on common-sense reasoning and sets a new state-of-the-art on the Long Range Arena (LRA) benchmark. Code is available at https://github.com/ZacharyMeng/NaLaFormer .
304	doi:10.1109/tcad.2024.3488572	2025	Improving Transformer Inference Through Optimized Nonlinear Operations With Quantization-Approximation-Based Strategy	Wen-Xun Wang	IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems	8	snowball-fwd-vit22b	1	SCREEN	exclude [E5] title/abstract: not about attention-logit control or LM training stability	Transformers have recently shown significant performance across various tasks, such as natural language processing (NLP) and computer vision (CV). However, the performance comes at the cost of large memory and computation overhead. Existing researches primarily focus on accelerating matrix multiplication (MatMul) through techniques like quantization and pruning, notably increasing the proportion of nonlinear operations in inference runtime. Meanwhile, previous approaches designed for nonlinear operations struggle with inefficient implementation as they are incapable of achieving both computation and memory efficiency. Additionally, these methods often require retraining or fine-tuning leading to substantial costs and inconveniences. To overcome these problems, we propose efficient implementation of nonlinear operations with quantization-approximation-based strategy. Through an in-depth analysis of the dataflow and data distribution of nonlinear operations, we design distinct quantization and approximation strategies tailored for different operations. Specifically, log2 quantization and power-of-two factor quantization have been employed in Softmax and LayerNorm, complemented by logarithmic function and low-precision statistic calculation as approximation strategies. Furthermore, the proposed efficient GeLU implementation integrates a nonuniform lookup procedure alongside low-bit-width quantization. Experimental results demonstrate negligible accuracy drops without the need for retraining or fine-tuning. By implementing the hardware design, it achieves $3.14\times - 6.34\times $ energy-efficiency and $3.01\times - 10.1\times $ area-efficiency improvements compared to state-of-the-art application-specific-integrated-circuit (ASIC) designs. In system-level evaluation, substantial speedup and reductions in energy consumption of 15% to 35% are achieved for end-to-end inference across both GPU and ASIC accelerator platforms.
305	arxiv:2502.17405	2025	Function-Space Learning Rates	Edward Milsom	International Conference on Machine Learning	6	snowball-fwd-vit22b	1	SCREEN	exclude [E5] title/abstract: not about attention-logit control or LM training stability	We consider layerwise function-space learning rates, which measure the magnitude of the change in a neural network's output function in response to an update to a parameter tensor. This contrasts with traditional learning rates, which describe the magnitude of changes in parameter space. We develop efficient methods to measure and set function-space learning rates in arbitrary neural networks, requiring only minimal computational overhead through a few additional backward passes that can be performed at the start of, or periodically during, training. We demonstrate two key applications: (1) analysing the dynamics of standard neural network optimisers in function space, rather than parameter space, and (2) introducing FLeRM (Function-space Learning Rate Matching), a novel approach to hyperparameter transfer across model scales. FLeRM records function-space learning rates while training a small, cheap base model, then automatically adjusts parameter-space layerwise learning rates when training larger models to maintain consistent function-space updates. FLeRM gives hyperparameter transfer across model width, depth, initialisation scale, and LoRA rank in various architectures including MLPs with residual connections and transformers with different layer normalisation schemes.
306	doi:10.1080/08839514.2024.2449296	2025	Discrete Wavelet Transform Sampling for Image Super Resolution	Chieh-Li Chen	Applied Artificial Intelligence	4	snowball-fwd-vit22b	1	SCREEN	exclude [E5] title/abstract: not about attention-logit control or LM training stability	ABSTRACT In battlefield environments, drones depend on high-resolution imagery for critical tasks such as target identification and situational awareness. However, acquiring clear images of distant targets presents a significant challenge. To address this, we propose a supervised learning approach for image super-resolution. Our network architecture builds upon the U-Net framework, incorporating enhancements to the encoder and decoder through techniques such as Discrete Wavelet Transform, Channel Attention Residual Modules, Selective Kernel Feature Fusion, Weight Normalization, and Dropout. We evaluate our model on a super-resolution dataset and compare its performance against other networks, highlighting the importance of minimizing trainable parameters for real-time deployment on resource-constrained drone platforms. The effWicacy of our proposed network is further validated through image recognition tasks and real-world scenario testing. By enhancing image clarity at extended ranges, our approach enables drones to detect adversaries earlier, facilitating proactive countermeasures and improving mission success rates
307	arxiv:2510.22026	2025	Normalization in Attention Dynamics	Nikita Karagodin	Neural Information Processing Systems	10	snowball-fwd-wortsman2024	1	SCREEN	maybe [IM] normalisation in attention dynamics	We study the effect of normalization schemes on token representations in deep transformers. Modeling their evolution as interacting particles on the sphere, we show that normalization acts as a form of speed regulation. This perspective enables a unified analysis of several schemes -- including Post-LN, Pre-LN, Mix-LN, Peri-LN, nGPT -- revealing how they influence clustering dynamics and representation collapse. Our framework clarifies how different schemes shape token representations across layers and provides a principled basis for comparing them, identifying Peri-LN as a particularly effective choice.
308	arxiv:2510.19093	2025	Weight Decay may matter more than muP for Learning Rate Transfer in Practice	Atli Kosson	arXiv.org	23	snowball-fwd-wortsman2024	1	SCREEN	maybe [BG] LR transfer without muP	Transferring the optimal learning rate from small to large neural networks can enable efficient training at scales where hyperparameter tuning is otherwise prohibitively expensive. To this end, the Maximal Update Parameterization (muP) proposes a learning rate scaling designed to keep the update dynamics of internal representations stable across different model widths. However, the scaling rules of muP rely on strong assumptions, particularly about the geometric alignment of a layer's inputs with both its weights and gradient updates. In this large-scale empirical investigation, we show that these assumptions hold only briefly at the start of training in the practical setups where learning rate transfer is most valuable, such as LLM training. For the remainder of training it is weight decay rather than muP that correctly stabilizes the update dynamics of internal representations across widths, facilitating learning rate transfer. This suggests muP's scaling primarily acts as a form of implicit learning rate warmup, allowing us to largely replace it with modified warmup schedules. Together these findings fundamentally challenge prevailing beliefs about learning rate transfer and can explain empirical observations such as why muP requires the independent weight decay variant for good transfer.
309	arxiv:2509.25087	2025	Scaling with Collapse: Efficient and Predictable Training of LLM Families	Shane Bergsma	arXiv.org	10	snowball-fwd-wortsman2024	1	SCREEN	exclude [E5] title/abstract: not about attention-logit control or LM training stability	Effective LLM training depends on predictable scaling of key quantities -- such as final loss and optimal hyperparameters -- with model and dataset size. Qiu et al. (2025) recently showed that this predictability can extend beyond scalars: whole training loss curves can collapse onto a universal trajectory after a simple normalization. What remains unclear is whether this phenomenon persists for LLM families trained under practical scaling recipes, where width, depth, learning rate, batch size, and weight decay are scaled jointly. We show that it does: loss curves collapse across scales precisely when optimization hyperparameters are set optimally for the given data budget, in accordance with recent empirical scaling laws. Collapse therefore emerges as a signature of compute-efficient training. We demonstrate two applications at scale: (1) deviation-from-collapse provides a sensitive, early diagnostic of training pathologies, and (2) predictability of collapsed curves enables early stopping in large-scale hyperparameter tuning. Finally, we train a competitive LLM family, Celerity, using these insights, establishing collapse as an effective tool for developing efficient LLMs.

The file is too large to be shown. View Raw

2.9 MiB Raw Blame History

2.9 MiB

Raw Blame History