Masters_Thesis/02-literature/extraction-table.csv

1	key	first_author	year	title	venue	ai_proposal	close_to_my_rq	model_sizes	data	question_asked	main_claim	method_varied_vs_fixed	qknorm_isolated	mup_used	lr_sweep_points_range	depth_varied	metrics	how_my_study_differs	notes
2	arxiv:2309.14322	Mitchell Wortsman	2023	Small-scale proxies for large-scale Transformer training instabilities	International Conference on Learning Representations	include [SEED*] LR-sensitivity metric; qk-layernorm at small scale	*
3	arxiv:2602.02522	George Grigorev	2026	IMU-1: Sample-Efficient Pre-training of Small Language Models	arXiv	include [SEED*] QK-norm largest win at 70M, bundled with muP	*
4	doi:10.18653/v1/2020.findings-emnlp.379	Alex Henry	2020	Query-Key Normalization for Transformers	Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP)	include [SEED] origin of QK-norm (ACL anthology copy of 2010.04245)
5	arxiv:2303.06296	Shuangfei Zhai	2023	Stabilizing Transformer Training by Preventing Attention Entropy Collapse	arXiv	include [SEED] attention entropy collapse <-> instability
6	arxiv:2302.05442	Mostafa Dehghani	2023	Scaling Vision Transformers to 22 Billion Parameters	International Conference on Machine Learning	include [SEED] QK-norm popularised to stop logit growth at 22B
7	arxiv:2410.16682	Rybakov, Oleg	2024	Methods of improving LLM training stability	arXiv (Cornell University)	include [IC*] QK-norm + softcap -> 1.5x LR before divergence, 830M	*
8	arxiv:2511.21377	Ben Anson	2025	Controlling changes to attention logits	arXiv	include [IC*] QK-norm vs alternatives for logit control; LR ablations	*
9	arxiv:2607.18363	Henry Ndubuaku	2026	A Controlled Study of Attention-Only Transformers	arXiv	include [IC*] QK-norm keeps 48-layer stacks trainable: depth evidence (H3)	*
10	arxiv:2610.34582	Harish Santhanalakshmi Ganesan	2026	HobbyLM: A Sparse Mixture-of-Experts Model Family on a Hobby Budget	arXiv	include [IS*] single-variable ablations at 130M incl. QK-norm (HobbyLM)	*
11	arxiv:2610.17969	Bum Jun Kim	2026	Neural Dimensional Analysis: Foundations of Dimensionless and Scale-Consistent Computation	arXiv	include [IA*] compares vs QK-Norm across 16x LR range at 124M	*
12	doi:10.5281/zenodo.20803082	Shubham Kumar	2026	Gating and Normalization Interact in the Attention Block	Zenodo (CERN European Organization for Nuclear Research)	include [IS*] 29-36M, QK-norm x depth x 3 seeds sweep; Zenodo preprint, check quality	*
13	arxiv:2606.16310	Yizhou Han	2026	QK-Normed MLA: QK normalization without full key caching	arXiv	include [IC] QK-norm permits higher LR; high-LR stress test vs QK-clip
14	arxiv:2603.02170	Jintao Zhang	2026	SageBwd: A Trainable Low-bit Attention	arXiv	include [IC] QK-norm necessary for stable low-bit training at large tokens/step
15	arxiv:2602.05006	Ezequiel López-Rubio	2026	Enhanced QKNorm normalization for neural transformers with the Lp norm	arXiv.org	include [IC] Lp-norm generalisation of QKNorm (variant)
16	arxiv:2503.09543	Oskar van der Wal	2025	PolyPythias: Stability and Outliers across Fifty Language Model Pre-Training Runs	International Conference on Learning Representations	include [IS] stability across 50 small-model seeds: seed-variance precedent
17	arxiv:2110.09456	Sam Shleifer	2021	NormFormer: Improved Transformer Pretraining with Extra Normalization	arXiv.org	include [IA] NormFormer: extra normalisation
18	arxiv:2312.16903	Sho Takase	2023	Spike No More: Stabilizing the Pre-training of Large Language Models	arXiv.org	include [IA] pretraining loss-spike mitigation
19	arxiv:2503.04598	Zhijian Zhuo	2025	HybridNorm: Towards Stable and Efficient Transformer Training via Hybrid Normalization	Neural Information Processing Systems	include [IA] HybridNorm: normalisation placement neighbour
20	arxiv:2502.02732	Jeonghoon Kim	2025	Peri-LN: Revisiting Normalization Layer in the Transformer Architecture	International Conference on Machine Learning	include [IA] Peri-LN: normalisation placement neighbour
21	arxiv:2507.20534	Kimi Team Yifan Bai	2025	Kimi K2: Open Agentic Intelligence		include [IA] MuonClip/QK-clip: alternative fix for attention-logit explosion
22	arxiv:2602.01212	Marco Chen	2026	SimpleGPT: Improving GPT via A Simple Normalization Strategy	arXiv.org	include [IA] SimpleGPT normalisation vs LLaMA2+QKNorm, LR ablations
23	arxiv:2304.13013	Mitchell Wortsman	2023	Stable and low-precision training for large-scale vision-language models	Neural Information Processing Systems	include [IM] Wortsman et al. follow-up on instability
24	arxiv:2410.10781	Xiangming Gu	2024	When Attention Sink Emerges in Language Models: An Empirical View	International Conference on Learning Representations	include [IM] when attention sinks emerge (seed paper ref)
25	arxiv:2402.17762	Mingjie Sun	2024	Massive Activations in Large Language Models	arXiv.org	include [IM] massive activations -> attention sinks
26	arxiv:2402.02098	Han Bao	2024	Self-attention Networks Localize When QK-eigenspectrum Concentrates	International Conference on Machine Learning	include [IM] QK eigenspectrum concentration and entropy collapse
27	arxiv:2505.15548	Suvadeep Hajra	2025	Dense Local Dependencies Induce Attention-Logit Explosion and Training Instability During Long-Sequence Transformer Training	arXiv	include [IM] dense local dependencies -> attention-logit explosion
28	arxiv:2505.21910	Xianbiao Qi	2025	Taming Transformer Without Using Learning Rate Warmup	arXiv	include [IM] W_q^T W_k spectral concentration -> entropy collapse, warmup
29	arxiv:2505.24333	Alessio Giorlandino	2025	Two failure modes of deep transformers and how to avoid them: a unified theory of signal propagation at initialisation	arXiv	include [IM] rank vs entropy collapse in deep transformers (depth)
30	doi:10.48448/nht8-e622	Association for Computational Linguistics 2025	2025	Variance Sensitivity Induces Attention Entropy Collapse and Instability in Transformers	Underline Science Inc.	include [IM] variance sensitivity -> attention entropy collapse
31	arxiv:2502.05795	Wenfang Sun	2025	The Curse of Depth in Large Language Models	Neural Information Processing Systems	include [IM] depth effects in LLMs (H3 context)
32	arxiv:2608.14712	Marios Papamichalis	2026	Which Question Is Your Attention Metric Answering? Attention Rows as Compositional Data		include [IM] how to measure attention entropy properly (metric choice)
33	arxiv:2002.04745	Ruibin Xiong	2020	On Layer Normalization in the Transformer Architecture	International Conference on Machine Learning	include [BG] Pre-LN vs Post-LN, warmup and LR (2020)
34	arxiv:2203.03466	Greg Yang	2022	Tensor Programs V: Tuning Large Neural Networks via Zero-Shot Hyperparameter Transfer	arXiv.org	include [BG] muP: the confound held OFF
35	arxiv:2501.18795	Bowen Yang	2025	Rope to Nope and Back Again: A New Hybrid Attention Strategy	arXiv	maybe [IC] RoPE/NoPE/QK-norm attention patterns, long context focus
36	arxiv:2603.11611	Mohammad Aflah Khan	2026	Fractional Rotation, Full Potential? Investigating Performance and Convergence of Partial RoPE	arXiv	maybe [IC] NoPE loss spikes mitigated by QK-norm (side finding)
37	arxiv:2608.30320	Zi-Han Qiu	2026	On the Design of Qwen3.8-Next Architecture: Evaluation, Efficiency, and Training Stability		maybe [IC] industry architecture study incl. training stability
38	arxiv:2609.14715	Dushyant Rajput	2026	Depth and Scale in the Sub-150M Regime: JugnuLM-53M vs JugnuLM-110M	arXiv	maybe [IS] sub-150M depth study (QK-norm fixed, not varied)
39	doi:10.5281/zenodo.18924690	Hang Lu	2026	Modernizing GPT-2-Scale Models Under a 10B-Token Training Budget	Zenodo (CERN European Organization for Nuclear Research)	maybe [IS] GPT-2-scale modernisation incl. QK-norm; Zenodo, check quality
40	arxiv:2507.13338	Laker Newhouse	2025	Training Transformers with Enforced Lipschitz Constants	arXiv	maybe [IA] Lipschitz transformers trained without QK-norm/softcap
41	arxiv:2505.22014	Jörg K. H. Franke	2025	Learning in Compact Spaces with Approximately Normalized Transformer	arXiv	maybe [IA] approx. normalised transformer vs GPT+QK-norm convergence
42	arxiv:2503.10622	Jia-Cheng Zhu	2025	Transformers without Normalization	Computer Vision and Pattern Recognition	maybe [IA] DyT replaces norm layers
43	arxiv:2601.02031	Felix Stollenwerk	2026	Output Embedding Centering for Stable LLM Pretraining	arXiv.org	maybe [IA] output-logit divergence; z-loss / softcap context
44	arxiv:2309.17453	Guang-Xuan Xiao	2023	Efficient Streaming Language Models with Attention Sinks	International Conference on Learning Representations	maybe [IM] attention sinks (inference paper, mechanism background)
45	arxiv:2410.13835	Tian-Yu Guo	2024	Active-Dormant Attention Heads: Mechanistically Demystifying Extreme-Token Phenomena in LLMs	arXiv.org	maybe [IM] extreme-token phenomena mechanism
46	arxiv:2501.12352	Ke Alexander Wang	2025	Test-time regression: a unifying framework for designing sequence models with associative memory	arXiv	maybe [IM] theory justifying QK-norm (test-time regression view)
47	arxiv:2502.00919	Stephen Zhang	2025	Attention Sinks: A 'Catch, Tag, Release' Mechanism for Embeddings	arXiv	maybe [IM] attention sinks persist with QK-norm
48	arxiv:2510.22026	Nikita Karagodin	2025	Normalization in Attention Dynamics	Neural Information Processing Systems	maybe [IM] normalisation in attention dynamics
49	arxiv:2608.02091	Shuxiao Xie	2026	One QK Channel, Many Sources: Tracing Low-Precision Attention Collapse	arXiv	maybe [IM] low-precision attention collapse; QK-norm disrupts it
50	arxiv:2609.34272	Jun-Lin Chen	2026	Broken Symmetry in BF16 Attention: Why FlashAttention Gradients Blow Up Late in Training		maybe [IM] BF16 attention gradients; logit growth late in training
51	arxiv:2606.28116	Rui-Xuan Huang	2026	Mechanism-Aligned Observability for Training Instability of Large Language Models		maybe [IM] instability diagnostics; possible metrics
52	arxiv:2601.22966	Zihan Qiu	2026	A Unified View of Attention and Residual Sinks: Outlier-Driven Rescaling is Essential for Transformer Training	arXiv (Cornell University)	maybe [IM] attention + residual sinks, outlier rescaling
53	arxiv:2602.18849	S. Emadi	2026	Exact Attention Sensitivity and the Geometry of Transformer Stability	arXiv.org	maybe [IM] attention sensitivity and stability geometry
54	arxiv:2304.03208	Nolan Dey	2023	Cerebras-GPT: Open Compute-Optimal Language Models Trained on the Cerebras Wafer-Scale Cluster	arXiv.org	maybe [BG] muP in small open models
55	arxiv:2407.07972	Rosie Zhao	2024	Deconstructing What Makes a Good Optimizer for Language Models	International Conference on Learning Representations	maybe [BG] LR stability across optimisers, small LMs
56	arxiv:2405.13698	Xi Wang	2024	How to set AdamW's weight decay as you scale model and dataset size	International Conference on Machine Learning	maybe [BG] AdamW hyperparameters vs scale
57	arxiv:2510.19093	Atli Kosson	2025	Weight Decay may matter more than muP for Learning Rate Transfer in Practice	arXiv.org	maybe [BG] LR transfer without muP
58	arxiv:2505.01618	Nolan Dey	2025	Don't be lazy: CompleteP enables compute-efficient deep transformers	Neural Information Processing Systems	maybe [BG] depth-wise parametrisation, LR across depth
59	arxiv:2608.24814	Zi-Han Liu	2026	Effective Learning Rate Governs Loss Dynamics in Language Model Pretraining		maybe [BG] effective LR view of loss dynamics