nik/Masters_Thesis
1
0 Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Add literature review protocol, searches and first snowball round

Browse Source 
Six seeds snowballed (Semantic Scholar) plus logged keyword searches
(OpenAlex, arXiv): 2,127 unique records, 480 title/abstract screened.
AI first-pass suggestions are in their own column; decisions are pending.

Claude-Session: https://claude.ai/code/session_01S8KN7NuUCm8nXLmy6eDuVv
This commit is contained in:
Cormac Shannon committed 2026-10-28 18:00:06 +00:00
1 parent 4369bcefa8
commit 95b5857189
28 files changed
+5547
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

02-literature/extraction-table.csv
+59
View File
@@ -0,0 +1,59 @@
key,first_author,year,title,venue,ai_proposal,close_to_my_rq,model_sizes,data,question_asked,main_claim,method_varied_vs_fixed,qknorm_isolated,mup_used,lr_sweep_points_range,depth_varied,metrics,how_my_study_differs,notes
arxiv:2309.14322,Mitchell Wortsman,2023,Small-scale proxies for large-scale Transformer training instabilities,International Conference on Learning Representations,include [SEED*] LR-sensitivity metric; qk-layernorm at small scale,*,,,,,,,,,,,,
arxiv:2602.02522,George Grigorev,2026,IMU-1: Sample-Efficient Pre-training of Small Language Models,arXiv,"include [SEED*] QK-norm largest win at 70M, bundled with muP",*,,,,,,,,,,,,
doi:10.18653/v1/2020.findings-emnlp.379,Alex Henry,2020,Query-Key Normalization for Transformers,Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP),include [SEED] origin of QK-norm (ACL anthology copy of 2010.04245),,,,,,,,,,,,,
arxiv:2303.06296,Shuangfei Zhai,2023,Stabilizing Transformer Training by Preventing Attention Entropy Collapse,arXiv,include [SEED] attention entropy collapse <-> instability,,,,,,,,,,,,,
arxiv:2302.05442,Mostafa Dehghani,2023,Scaling Vision Transformers to 22 Billion Parameters,International Conference on Machine Learning,include [SEED] QK-norm popularised to stop logit growth at 22B,,,,,,,,,,,,,
arxiv:2410.16682,"Rybakov, Oleg",2024,Methods of improving LLM training stability,arXiv (Cornell University),"include [IC*] QK-norm + softcap -> 1.5x LR before divergence, 830M",*,,,,,,,,,,,,
arxiv:2511.21377,Ben Anson,2025,Controlling changes to attention logits,arXiv,include [IC*] QK-norm vs alternatives for logit control; LR ablations,*,,,,,,,,,,,,
arxiv:2607.18363,Henry Ndubuaku,2026,A Controlled Study of Attention-Only Transformers,arXiv,include [IC*] QK-norm keeps 48-layer stacks trainable: depth evidence (H3),*,,,,,,,,,,,,
arxiv:2610.34582,Harish Santhanalakshmi Ganesan,2026,HobbyLM: A Sparse Mixture-of-Experts Model Family on a Hobby Budget,arXiv,include [IS*] single-variable ablations at 130M incl. QK-norm (HobbyLM),*,,,,,,,,,,,,
arxiv:2610.17969,Bum Jun Kim,2026,Neural Dimensional Analysis: Foundations of Dimensionless and Scale-Consistent Computation,arXiv,include [IA*] compares vs QK-Norm across 16x LR range at 124M,*,,,,,,,,,,,,
doi:10.5281/zenodo.20803082,Shubham Kumar,2026,Gating and Normalization Interact in the Attention Block,Zenodo (CERN European Organization for Nuclear Research),"include [IS*] 29-36M, QK-norm x depth x 3 seeds sweep; Zenodo preprint, check quality",*,,,,,,,,,,,,
arxiv:2606.16310,Yizhou Han,2026,QK-Normed MLA: QK normalization without full key caching,arXiv,include [IC] QK-norm permits higher LR; high-LR stress test vs QK-clip,,,,,,,,,,,,,
arxiv:2603.02170,Jintao Zhang,2026,SageBwd: A Trainable Low-bit Attention,arXiv,include [IC] QK-norm necessary for stable low-bit training at large tokens/step,,,,,,,,,,,,,
arxiv:2602.05006,Ezequiel López-Rubio,2026,Enhanced QKNorm normalization for neural transformers with the Lp norm,arXiv.org,include [IC] Lp-norm generalisation of QKNorm (variant),,,,,,,,,,,,,
arxiv:2503.09543,Oskar van der Wal,2025,PolyPythias: Stability and Outliers across Fifty Language Model Pre-Training Runs,International Conference on Learning Representations,include [IS] stability across 50 small-model seeds: seed-variance precedent,,,,,,,,,,,,,
arxiv:2110.09456,Sam Shleifer,2021,NormFormer: Improved Transformer Pretraining with Extra Normalization,arXiv.org,include [IA] NormFormer: extra normalisation,,,,,,,,,,,,,
arxiv:2312.16903,Sho Takase,2023,Spike No More: Stabilizing the Pre-training of Large Language Models,arXiv.org,include [IA] pretraining loss-spike mitigation,,,,,,,,,,,,,
arxiv:2503.04598,Zhijian Zhuo,2025,HybridNorm: Towards Stable and Efficient Transformer Training via Hybrid Normalization,Neural Information Processing Systems,include [IA] HybridNorm: normalisation placement neighbour,,,,,,,,,,,,,
arxiv:2502.02732,Jeonghoon Kim,2025,Peri-LN: Revisiting Normalization Layer in the Transformer Architecture,International Conference on Machine Learning,include [IA] Peri-LN: normalisation placement neighbour,,,,,,,,,,,,,
arxiv:2507.20534,Kimi Team Yifan Bai,2025,Kimi K2: Open Agentic Intelligence,,include [IA] MuonClip/QK-clip: alternative fix for attention-logit explosion,,,,,,,,,,,,,
arxiv:2602.01212,Marco Chen,2026,SimpleGPT: Improving GPT via A Simple Normalization Strategy,arXiv.org,"include [IA] SimpleGPT normalisation vs LLaMA2+QKNorm, LR ablations",,,,,,,,,,,,,
arxiv:2304.13013,Mitchell Wortsman,2023,Stable and low-precision training for large-scale vision-language models,Neural Information Processing Systems,include [IM] Wortsman et al. follow-up on instability,,,,,,,,,,,,,
arxiv:2410.10781,Xiangming Gu,2024,When Attention Sink Emerges in Language Models: An Empirical View,International Conference on Learning Representations,include [IM] when attention sinks emerge (seed paper ref),,,,,,,,,,,,,
arxiv:2402.17762,Mingjie Sun,2024,Massive Activations in Large Language Models,arXiv.org,include [IM] massive activations -> attention sinks,,,,,,,,,,,,,
arxiv:2402.02098,Han Bao,2024,Self-attention Networks Localize When QK-eigenspectrum Concentrates,International Conference on Machine Learning,include [IM] QK eigenspectrum concentration and entropy collapse,,,,,,,,,,,,,
arxiv:2505.15548,Suvadeep Hajra,2025,Dense Local Dependencies Induce Attention-Logit Explosion and Training Instability During Long-Sequence Transformer Training,arXiv,include [IM] dense local dependencies -> attention-logit explosion,,,,,,,,,,,,,
arxiv:2505.21910,Xianbiao Qi,2025,Taming Transformer Without Using Learning Rate Warmup,arXiv,"include [IM] W_q^T W_k spectral concentration -> entropy collapse, warmup",,,,,,,,,,,,,
arxiv:2505.24333,Alessio Giorlandino,2025,Two failure modes of deep transformers and how to avoid them: a unified theory of signal propagation at initialisation,arXiv,include [IM] rank vs entropy collapse in deep transformers (depth),,,,,,,,,,,,,
doi:10.48448/nht8-e622,Association for Computational Linguistics 2025,2025,Variance Sensitivity Induces Attention Entropy Collapse and Instability in Transformers,Underline Science Inc.,include [IM] variance sensitivity -> attention entropy collapse,,,,,,,,,,,,,
arxiv:2502.05795,Wenfang Sun,2025,The Curse of Depth in Large Language Models,Neural Information Processing Systems,include [IM] depth effects in LLMs (H3 context),,,,,,,,,,,,,
arxiv:2608.14712,Marios Papamichalis,2026,Which Question Is Your Attention Metric Answering? Attention Rows as Compositional Data,,include [IM] how to measure attention entropy properly (metric choice),,,,,,,,,,,,,
arxiv:2002.04745,Ruibin Xiong,2020,On Layer Normalization in the Transformer Architecture,International Conference on Machine Learning,"include [BG] Pre-LN vs Post-LN, warmup and LR (2020)",,,,,,,,,,,,,
arxiv:2203.03466,Greg Yang,2022,Tensor Programs V: Tuning Large Neural Networks via Zero-Shot Hyperparameter Transfer,arXiv.org,include [BG] muP: the confound held OFF,,,,,,,,,,,,,
arxiv:2501.18795,Bowen Yang,2025,Rope to Nope and Back Again: A New Hybrid Attention Strategy,arXiv,"maybe [IC] RoPE/NoPE/QK-norm attention patterns, long context focus",,,,,,,,,,,,,
arxiv:2603.11611,Mohammad Aflah Khan,2026,"Fractional Rotation, Full Potential? Investigating Performance and Convergence of Partial RoPE",arXiv,maybe [IC] NoPE loss spikes mitigated by QK-norm (side finding),,,,,,,,,,,,,
arxiv:2608.30320,Zi-Han Qiu,2026,"On the Design of Qwen3.8-Next Architecture: Evaluation, Efficiency, and Training Stability",,maybe [IC] industry architecture study incl. training stability,,,,,,,,,,,,,
arxiv:2609.14715,Dushyant Rajput,2026,Depth and Scale in the Sub-150M Regime: JugnuLM-53M vs JugnuLM-110M,arXiv,"maybe [IS] sub-150M depth study (QK-norm fixed, not varied)",,,,,,,,,,,,,
doi:10.5281/zenodo.18924690,Hang Lu,2026,Modernizing GPT-2-Scale Models Under a 10B-Token Training Budget,Zenodo (CERN European Organization for Nuclear Research),"maybe [IS] GPT-2-scale modernisation incl. QK-norm; Zenodo, check quality",,,,,,,,,,,,,
arxiv:2507.13338,Laker Newhouse,2025,Training Transformers with Enforced Lipschitz Constants,arXiv,maybe [IA] Lipschitz transformers trained without QK-norm/softcap,,,,,,,,,,,,,
arxiv:2505.22014,Jörg K. H. Franke,2025,Learning in Compact Spaces with Approximately Normalized Transformer,arXiv,maybe [IA] approx. normalised transformer vs GPT+QK-norm convergence,,,,,,,,,,,,,
arxiv:2503.10622,Jia-Cheng Zhu,2025,Transformers without Normalization,Computer Vision and Pattern Recognition,maybe [IA] DyT replaces norm layers,,,,,,,,,,,,,
arxiv:2601.02031,Felix Stollenwerk,2026,Output Embedding Centering for Stable LLM Pretraining,arXiv.org,maybe [IA] output-logit divergence; z-loss / softcap context,,,,,,,,,,,,,
arxiv:2309.17453,Guang-Xuan Xiao,2023,Efficient Streaming Language Models with Attention Sinks,International Conference on Learning Representations,"maybe [IM] attention sinks (inference paper, mechanism background)",,,,,,,,,,,,,
arxiv:2410.13835,Tian-Yu Guo,2024,Active-Dormant Attention Heads: Mechanistically Demystifying Extreme-Token Phenomena in LLMs,arXiv.org,maybe [IM] extreme-token phenomena mechanism,,,,,,,,,,,,,
arxiv:2501.12352,Ke Alexander Wang,2025,Test-time regression: a unifying framework for designing sequence models with associative memory,arXiv,maybe [IM] theory justifying QK-norm (test-time regression view),,,,,,,,,,,,,
arxiv:2502.00919,Stephen Zhang,2025,"Attention Sinks: A 'Catch, Tag, Release' Mechanism for Embeddings",arXiv,maybe [IM] attention sinks persist with QK-norm,,,,,,,,,,,,,
arxiv:2510.22026,Nikita Karagodin,2025,Normalization in Attention Dynamics,Neural Information Processing Systems,maybe [IM] normalisation in attention dynamics,,,,,,,,,,,,,
arxiv:2608.02091,Shuxiao Xie,2026,"One QK Channel, Many Sources: Tracing Low-Precision Attention Collapse",arXiv,maybe [IM] low-precision attention collapse; QK-norm disrupts it,,,,,,,,,,,,,
arxiv:2609.34272,Jun-Lin Chen,2026,Broken Symmetry in BF16 Attention: Why FlashAttention Gradients Blow Up Late in Training,,maybe [IM] BF16 attention gradients; logit growth late in training,,,,,,,,,,,,,
arxiv:2606.28116,Rui-Xuan Huang,2026,Mechanism-Aligned Observability for Training Instability of Large Language Models,,maybe [IM] instability diagnostics; possible metrics,,,,,,,,,,,,,
arxiv:2601.22966,Zihan Qiu,2026,A Unified View of Attention and Residual Sinks: Outlier-Driven Rescaling is Essential for Transformer Training,arXiv (Cornell University),"maybe [IM] attention + residual sinks, outlier rescaling",,,,,,,,,,,,,
arxiv:2602.18849,S. Emadi,2026,Exact Attention Sensitivity and the Geometry of Transformer Stability,arXiv.org,maybe [IM] attention sensitivity and stability geometry,,,,,,,,,,,,,
arxiv:2304.03208,Nolan Dey,2023,Cerebras-GPT: Open Compute-Optimal Language Models Trained on the Cerebras Wafer-Scale Cluster,arXiv.org,maybe [BG] muP in small open models,,,,,,,,,,,,,
arxiv:2407.07972,Rosie Zhao,2024,Deconstructing What Makes a Good Optimizer for Language Models,International Conference on Learning Representations,"maybe [BG] LR stability across optimisers, small LMs",,,,,,,,,,,,,
arxiv:2405.13698,Xi Wang,2024,How to set AdamW's weight decay as you scale model and dataset size,International Conference on Machine Learning,maybe [BG] AdamW hyperparameters vs scale,,,,,,,,,,,,,
arxiv:2510.19093,Atli Kosson,2025,Weight Decay may matter more than muP for Learning Rate Transfer in Practice,arXiv.org,maybe [BG] LR transfer without muP,,,,,,,,,,,,,
arxiv:2505.01618,Nolan Dey,2025,Don't be lazy: CompleteP enables compute-efficient deep transformers,Neural Information Processing Systems,"maybe [BG] depth-wise parametrisation, LR across depth",,,,,,,,,,,,,
arxiv:2608.24814,Zi-Han Liu,2026,Effective Learning Rate Governs Loss Dynamics in Language Model Pretraining,,maybe [BG] effective LR view of loss dynamics,,,,,,,,,,,,,
1 key first_author year title venue ai_proposal close_to_my_rq model_sizes data question_asked main_claim method_varied_vs_fixed qknorm_isolated mup_used lr_sweep_points_range depth_varied metrics how_my_study_differs notes
2 arxiv:2309.14322 Mitchell Wortsman 2023 Small-scale proxies for large-scale Transformer training instabilities International Conference on Learning Representations include [SEED*] LR-sensitivity metric; qk-layernorm at small scale *
3 arxiv:2602.02522 George Grigorev 2026 IMU-1: Sample-Efficient Pre-training of Small Language Models arXiv include [SEED*] QK-norm largest win at 70M, bundled with muP *
4 doi:10.18653/v1/2020.findings-emnlp.379 Alex Henry 2020 Query-Key Normalization for Transformers Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) include [SEED] origin of QK-norm (ACL anthology copy of 2010.04245)
5 arxiv:2303.06296 Shuangfei Zhai 2023 Stabilizing Transformer Training by Preventing Attention Entropy Collapse arXiv include [SEED] attention entropy collapse <-> instability
6 arxiv:2302.05442 Mostafa Dehghani 2023 Scaling Vision Transformers to 22 Billion Parameters International Conference on Machine Learning include [SEED] QK-norm popularised to stop logit growth at 22B
7 arxiv:2410.16682 Rybakov, Oleg 2024 Methods of improving LLM training stability arXiv (Cornell University) include [IC*] QK-norm + softcap -> 1.5x LR before divergence, 830M *
8 arxiv:2511.21377 Ben Anson 2025 Controlling changes to attention logits arXiv include [IC*] QK-norm vs alternatives for logit control; LR ablations *
9 arxiv:2607.18363 Henry Ndubuaku 2026 A Controlled Study of Attention-Only Transformers arXiv include [IC*] QK-norm keeps 48-layer stacks trainable: depth evidence (H3) *
10 arxiv:2610.34582 Harish Santhanalakshmi Ganesan 2026 HobbyLM: A Sparse Mixture-of-Experts Model Family on a Hobby Budget arXiv include [IS*] single-variable ablations at 130M incl. QK-norm (HobbyLM) *
11 arxiv:2610.17969 Bum Jun Kim 2026 Neural Dimensional Analysis: Foundations of Dimensionless and Scale-Consistent Computation arXiv include [IA*] compares vs QK-Norm across 16x LR range at 124M *
12 doi:10.5281/zenodo.20803082 Shubham Kumar 2026 Gating and Normalization Interact in the Attention Block Zenodo (CERN European Organization for Nuclear Research) include [IS*] 29-36M, QK-norm x depth x 3 seeds sweep; Zenodo preprint, check quality *
13 arxiv:2606.16310 Yizhou Han 2026 QK-Normed MLA: QK normalization without full key caching arXiv include [IC] QK-norm permits higher LR; high-LR stress test vs QK-clip
14 arxiv:2603.02170 Jintao Zhang 2026 SageBwd: A Trainable Low-bit Attention arXiv include [IC] QK-norm necessary for stable low-bit training at large tokens/step
15 arxiv:2602.05006 Ezequiel López-Rubio 2026 Enhanced QKNorm normalization for neural transformers with the Lp norm arXiv.org include [IC] Lp-norm generalisation of QKNorm (variant)
16 arxiv:2503.09543 Oskar van der Wal 2025 PolyPythias: Stability and Outliers across Fifty Language Model Pre-Training Runs International Conference on Learning Representations include [IS] stability across 50 small-model seeds: seed-variance precedent
17 arxiv:2110.09456 Sam Shleifer 2021 NormFormer: Improved Transformer Pretraining with Extra Normalization arXiv.org include [IA] NormFormer: extra normalisation
18 arxiv:2312.16903 Sho Takase 2023 Spike No More: Stabilizing the Pre-training of Large Language Models arXiv.org include [IA] pretraining loss-spike mitigation
19 arxiv:2503.04598 Zhijian Zhuo 2025 HybridNorm: Towards Stable and Efficient Transformer Training via Hybrid Normalization Neural Information Processing Systems include [IA] HybridNorm: normalisation placement neighbour
20 arxiv:2502.02732 Jeonghoon Kim 2025 Peri-LN: Revisiting Normalization Layer in the Transformer Architecture International Conference on Machine Learning include [IA] Peri-LN: normalisation placement neighbour
21 arxiv:2507.20534 Kimi Team Yifan Bai 2025 Kimi K2: Open Agentic Intelligence include [IA] MuonClip/QK-clip: alternative fix for attention-logit explosion
22 arxiv:2602.01212 Marco Chen 2026 SimpleGPT: Improving GPT via A Simple Normalization Strategy arXiv.org include [IA] SimpleGPT normalisation vs LLaMA2+QKNorm, LR ablations
23 arxiv:2304.13013 Mitchell Wortsman 2023 Stable and low-precision training for large-scale vision-language models Neural Information Processing Systems include [IM] Wortsman et al. follow-up on instability
24 arxiv:2410.10781 Xiangming Gu 2024 When Attention Sink Emerges in Language Models: An Empirical View International Conference on Learning Representations include [IM] when attention sinks emerge (seed paper ref)
25 arxiv:2402.17762 Mingjie Sun 2024 Massive Activations in Large Language Models arXiv.org include [IM] massive activations -> attention sinks
26 arxiv:2402.02098 Han Bao 2024 Self-attention Networks Localize When QK-eigenspectrum Concentrates International Conference on Machine Learning include [IM] QK eigenspectrum concentration and entropy collapse
27 arxiv:2505.15548 Suvadeep Hajra 2025 Dense Local Dependencies Induce Attention-Logit Explosion and Training Instability During Long-Sequence Transformer Training arXiv include [IM] dense local dependencies -> attention-logit explosion
28 arxiv:2505.21910 Xianbiao Qi 2025 Taming Transformer Without Using Learning Rate Warmup arXiv include [IM] W_q^T W_k spectral concentration -> entropy collapse, warmup
29 arxiv:2505.24333 Alessio Giorlandino 2025 Two failure modes of deep transformers and how to avoid them: a unified theory of signal propagation at initialisation arXiv include [IM] rank vs entropy collapse in deep transformers (depth)
30 doi:10.48448/nht8-e622 Association for Computational Linguistics 2025 2025 Variance Sensitivity Induces Attention Entropy Collapse and Instability in Transformers Underline Science Inc. include [IM] variance sensitivity -> attention entropy collapse
31 arxiv:2502.05795 Wenfang Sun 2025 The Curse of Depth in Large Language Models Neural Information Processing Systems include [IM] depth effects in LLMs (H3 context)
32 arxiv:2608.14712 Marios Papamichalis 2026 Which Question Is Your Attention Metric Answering? Attention Rows as Compositional Data include [IM] how to measure attention entropy properly (metric choice)
33 arxiv:2002.04745 Ruibin Xiong 2020 On Layer Normalization in the Transformer Architecture International Conference on Machine Learning include [BG] Pre-LN vs Post-LN, warmup and LR (2020)
34 arxiv:2203.03466 Greg Yang 2022 Tensor Programs V: Tuning Large Neural Networks via Zero-Shot Hyperparameter Transfer arXiv.org include [BG] muP: the confound held OFF
35 arxiv:2501.18795 Bowen Yang 2025 Rope to Nope and Back Again: A New Hybrid Attention Strategy arXiv maybe [IC] RoPE/NoPE/QK-norm attention patterns, long context focus
36 arxiv:2603.11611 Mohammad Aflah Khan 2026 Fractional Rotation, Full Potential? Investigating Performance and Convergence of Partial RoPE arXiv maybe [IC] NoPE loss spikes mitigated by QK-norm (side finding)
37 arxiv:2608.30320 Zi-Han Qiu 2026 On the Design of Qwen3.8-Next Architecture: Evaluation, Efficiency, and Training Stability maybe [IC] industry architecture study incl. training stability
38 arxiv:2609.14715 Dushyant Rajput 2026 Depth and Scale in the Sub-150M Regime: JugnuLM-53M vs JugnuLM-110M arXiv maybe [IS] sub-150M depth study (QK-norm fixed, not varied)
39 doi:10.5281/zenodo.18924690 Hang Lu 2026 Modernizing GPT-2-Scale Models Under a 10B-Token Training Budget Zenodo (CERN European Organization for Nuclear Research) maybe [IS] GPT-2-scale modernisation incl. QK-norm; Zenodo, check quality
40 arxiv:2507.13338 Laker Newhouse 2025 Training Transformers with Enforced Lipschitz Constants arXiv maybe [IA] Lipschitz transformers trained without QK-norm/softcap
41 arxiv:2505.22014 Jörg K. H. Franke 2025 Learning in Compact Spaces with Approximately Normalized Transformer arXiv maybe [IA] approx. normalised transformer vs GPT+QK-norm convergence
42 arxiv:2503.10622 Jia-Cheng Zhu 2025 Transformers without Normalization Computer Vision and Pattern Recognition maybe [IA] DyT replaces norm layers
43 arxiv:2601.02031 Felix Stollenwerk 2026 Output Embedding Centering for Stable LLM Pretraining arXiv.org maybe [IA] output-logit divergence; z-loss / softcap context
44 arxiv:2309.17453 Guang-Xuan Xiao 2023 Efficient Streaming Language Models with Attention Sinks International Conference on Learning Representations maybe [IM] attention sinks (inference paper, mechanism background)
45 arxiv:2410.13835 Tian-Yu Guo 2024 Active-Dormant Attention Heads: Mechanistically Demystifying Extreme-Token Phenomena in LLMs arXiv.org maybe [IM] extreme-token phenomena mechanism
46 arxiv:2501.12352 Ke Alexander Wang 2025 Test-time regression: a unifying framework for designing sequence models with associative memory arXiv maybe [IM] theory justifying QK-norm (test-time regression view)
47 arxiv:2502.00919 Stephen Zhang 2025 Attention Sinks: A 'Catch, Tag, Release' Mechanism for Embeddings arXiv maybe [IM] attention sinks persist with QK-norm
48 arxiv:2510.22026 Nikita Karagodin 2025 Normalization in Attention Dynamics Neural Information Processing Systems maybe [IM] normalisation in attention dynamics
49 arxiv:2608.02091 Shuxiao Xie 2026 One QK Channel, Many Sources: Tracing Low-Precision Attention Collapse arXiv maybe [IM] low-precision attention collapse; QK-norm disrupts it
50 arxiv:2609.34272 Jun-Lin Chen 2026 Broken Symmetry in BF16 Attention: Why FlashAttention Gradients Blow Up Late in Training maybe [IM] BF16 attention gradients; logit growth late in training
51 arxiv:2606.28116 Rui-Xuan Huang 2026 Mechanism-Aligned Observability for Training Instability of Large Language Models maybe [IM] instability diagnostics; possible metrics
52 arxiv:2601.22966 Zihan Qiu 2026 A Unified View of Attention and Residual Sinks: Outlier-Driven Rescaling is Essential for Transformer Training arXiv (Cornell University) maybe [IM] attention + residual sinks, outlier rescaling
53 arxiv:2602.18849 S. Emadi 2026 Exact Attention Sensitivity and the Geometry of Transformer Stability arXiv.org maybe [IM] attention sensitivity and stability geometry
54 arxiv:2304.03208 Nolan Dey 2023 Cerebras-GPT: Open Compute-Optimal Language Models Trained on the Cerebras Wafer-Scale Cluster arXiv.org maybe [BG] muP in small open models
55 arxiv:2407.07972 Rosie Zhao 2024 Deconstructing What Makes a Good Optimizer for Language Models International Conference on Learning Representations maybe [BG] LR stability across optimisers, small LMs
56 arxiv:2405.13698 Xi Wang 2024 How to set AdamW's weight decay as you scale model and dataset size International Conference on Machine Learning maybe [BG] AdamW hyperparameters vs scale
57 arxiv:2510.19093 Atli Kosson 2025 Weight Decay may matter more than muP for Learning Rate Transfer in Practice arXiv.org maybe [BG] LR transfer without muP
58 arxiv:2505.01618 Nolan Dey 2025 Don't be lazy: CompleteP enables compute-efficient deep transformers Neural Information Processing Systems maybe [BG] depth-wise parametrisation, LR across depth
59 arxiv:2608.24814 Zi-Han Liu 2026 Effective Learning Rate Governs Loss Dynamics in Language Model Pretraining maybe [BG] effective LR view of loss dynamics