Cormac Shannon
95b5857189
Six seeds snowballed (Semantic Scholar) plus logged keyword searches (OpenAlex, arXiv): 2,127 unique records, 480 title/abstract screened. AI first-pass suggestions are in their own column; decisions are pending. Claude-Session: https://claude.ai/code/session_01S8KN7NuUCm8nXLmy6eDuVv
60 lines
12 KiB
CSV
60 lines
12 KiB
CSV
key,first_author,year,title,venue,ai_proposal,close_to_my_rq,model_sizes,data,question_asked,main_claim,method_varied_vs_fixed,qknorm_isolated,mup_used,lr_sweep_points_range,depth_varied,metrics,how_my_study_differs,notes
|
|
arxiv:2309.14322,Mitchell Wortsman,2023,Small-scale proxies for large-scale Transformer training instabilities,International Conference on Learning Representations,include [SEED*] LR-sensitivity metric; qk-layernorm at small scale,*,,,,,,,,,,,,
|
|
arxiv:2602.02522,George Grigorev,2026,IMU-1: Sample-Efficient Pre-training of Small Language Models,arXiv,"include [SEED*] QK-norm largest win at 70M, bundled with muP",*,,,,,,,,,,,,
|
|
doi:10.18653/v1/2020.findings-emnlp.379,Alex Henry,2020,Query-Key Normalization for Transformers,Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP),include [SEED] origin of QK-norm (ACL anthology copy of 2010.04245),,,,,,,,,,,,,
|
|
arxiv:2303.06296,Shuangfei Zhai,2023,Stabilizing Transformer Training by Preventing Attention Entropy Collapse,arXiv,include [SEED] attention entropy collapse <-> instability,,,,,,,,,,,,,
|
|
arxiv:2302.05442,Mostafa Dehghani,2023,Scaling Vision Transformers to 22 Billion Parameters,International Conference on Machine Learning,include [SEED] QK-norm popularised to stop logit growth at 22B,,,,,,,,,,,,,
|
|
arxiv:2410.16682,"Rybakov, Oleg",2024,Methods of improving LLM training stability,arXiv (Cornell University),"include [IC*] QK-norm + softcap -> 1.5x LR before divergence, 830M",*,,,,,,,,,,,,
|
|
arxiv:2511.21377,Ben Anson,2025,Controlling changes to attention logits,arXiv,include [IC*] QK-norm vs alternatives for logit control; LR ablations,*,,,,,,,,,,,,
|
|
arxiv:2607.18363,Henry Ndubuaku,2026,A Controlled Study of Attention-Only Transformers,arXiv,include [IC*] QK-norm keeps 48-layer stacks trainable: depth evidence (H3),*,,,,,,,,,,,,
|
|
arxiv:2610.34582,Harish Santhanalakshmi Ganesan,2026,HobbyLM: A Sparse Mixture-of-Experts Model Family on a Hobby Budget,arXiv,include [IS*] single-variable ablations at 130M incl. QK-norm (HobbyLM),*,,,,,,,,,,,,
|
|
arxiv:2610.17969,Bum Jun Kim,2026,Neural Dimensional Analysis: Foundations of Dimensionless and Scale-Consistent Computation,arXiv,include [IA*] compares vs QK-Norm across 16x LR range at 124M,*,,,,,,,,,,,,
|
|
doi:10.5281/zenodo.20803082,Shubham Kumar,2026,Gating and Normalization Interact in the Attention Block,Zenodo (CERN European Organization for Nuclear Research),"include [IS*] 29-36M, QK-norm x depth x 3 seeds sweep; Zenodo preprint, check quality",*,,,,,,,,,,,,
|
|
arxiv:2606.16310,Yizhou Han,2026,QK-Normed MLA: QK normalization without full key caching,arXiv,include [IC] QK-norm permits higher LR; high-LR stress test vs QK-clip,,,,,,,,,,,,,
|
|
arxiv:2603.02170,Jintao Zhang,2026,SageBwd: A Trainable Low-bit Attention,arXiv,include [IC] QK-norm necessary for stable low-bit training at large tokens/step,,,,,,,,,,,,,
|
|
arxiv:2602.05006,Ezequiel López-Rubio,2026,Enhanced QKNorm normalization for neural transformers with the Lp norm,arXiv.org,include [IC] Lp-norm generalisation of QKNorm (variant),,,,,,,,,,,,,
|
|
arxiv:2503.09543,Oskar van der Wal,2025,PolyPythias: Stability and Outliers across Fifty Language Model Pre-Training Runs,International Conference on Learning Representations,include [IS] stability across 50 small-model seeds: seed-variance precedent,,,,,,,,,,,,,
|
|
arxiv:2110.09456,Sam Shleifer,2021,NormFormer: Improved Transformer Pretraining with Extra Normalization,arXiv.org,include [IA] NormFormer: extra normalisation,,,,,,,,,,,,,
|
|
arxiv:2312.16903,Sho Takase,2023,Spike No More: Stabilizing the Pre-training of Large Language Models,arXiv.org,include [IA] pretraining loss-spike mitigation,,,,,,,,,,,,,
|
|
arxiv:2503.04598,Zhijian Zhuo,2025,HybridNorm: Towards Stable and Efficient Transformer Training via Hybrid Normalization,Neural Information Processing Systems,include [IA] HybridNorm: normalisation placement neighbour,,,,,,,,,,,,,
|
|
arxiv:2502.02732,Jeonghoon Kim,2025,Peri-LN: Revisiting Normalization Layer in the Transformer Architecture,International Conference on Machine Learning,include [IA] Peri-LN: normalisation placement neighbour,,,,,,,,,,,,,
|
|
arxiv:2507.20534,Kimi Team Yifan Bai,2025,Kimi K2: Open Agentic Intelligence,,include [IA] MuonClip/QK-clip: alternative fix for attention-logit explosion,,,,,,,,,,,,,
|
|
arxiv:2602.01212,Marco Chen,2026,SimpleGPT: Improving GPT via A Simple Normalization Strategy,arXiv.org,"include [IA] SimpleGPT normalisation vs LLaMA2+QKNorm, LR ablations",,,,,,,,,,,,,
|
|
arxiv:2304.13013,Mitchell Wortsman,2023,Stable and low-precision training for large-scale vision-language models,Neural Information Processing Systems,include [IM] Wortsman et al. follow-up on instability,,,,,,,,,,,,,
|
|
arxiv:2410.10781,Xiangming Gu,2024,When Attention Sink Emerges in Language Models: An Empirical View,International Conference on Learning Representations,include [IM] when attention sinks emerge (seed paper ref),,,,,,,,,,,,,
|
|
arxiv:2402.17762,Mingjie Sun,2024,Massive Activations in Large Language Models,arXiv.org,include [IM] massive activations -> attention sinks,,,,,,,,,,,,,
|
|
arxiv:2402.02098,Han Bao,2024,Self-attention Networks Localize When QK-eigenspectrum Concentrates,International Conference on Machine Learning,include [IM] QK eigenspectrum concentration and entropy collapse,,,,,,,,,,,,,
|
|
arxiv:2505.15548,Suvadeep Hajra,2025,Dense Local Dependencies Induce Attention-Logit Explosion and Training Instability During Long-Sequence Transformer Training,arXiv,include [IM] dense local dependencies -> attention-logit explosion,,,,,,,,,,,,,
|
|
arxiv:2505.21910,Xianbiao Qi,2025,Taming Transformer Without Using Learning Rate Warmup,arXiv,"include [IM] W_q^T W_k spectral concentration -> entropy collapse, warmup",,,,,,,,,,,,,
|
|
arxiv:2505.24333,Alessio Giorlandino,2025,Two failure modes of deep transformers and how to avoid them: a unified theory of signal propagation at initialisation,arXiv,include [IM] rank vs entropy collapse in deep transformers (depth),,,,,,,,,,,,,
|
|
doi:10.48448/nht8-e622,Association for Computational Linguistics 2025,2025,Variance Sensitivity Induces Attention Entropy Collapse and Instability in Transformers,Underline Science Inc.,include [IM] variance sensitivity -> attention entropy collapse,,,,,,,,,,,,,
|
|
arxiv:2502.05795,Wenfang Sun,2025,The Curse of Depth in Large Language Models,Neural Information Processing Systems,include [IM] depth effects in LLMs (H3 context),,,,,,,,,,,,,
|
|
arxiv:2608.14712,Marios Papamichalis,2026,Which Question Is Your Attention Metric Answering? Attention Rows as Compositional Data,,include [IM] how to measure attention entropy properly (metric choice),,,,,,,,,,,,,
|
|
arxiv:2002.04745,Ruibin Xiong,2020,On Layer Normalization in the Transformer Architecture,International Conference on Machine Learning,"include [BG] Pre-LN vs Post-LN, warmup and LR (2020)",,,,,,,,,,,,,
|
|
arxiv:2203.03466,Greg Yang,2022,Tensor Programs V: Tuning Large Neural Networks via Zero-Shot Hyperparameter Transfer,arXiv.org,include [BG] muP: the confound held OFF,,,,,,,,,,,,,
|
|
arxiv:2501.18795,Bowen Yang,2025,Rope to Nope and Back Again: A New Hybrid Attention Strategy,arXiv,"maybe [IC] RoPE/NoPE/QK-norm attention patterns, long context focus",,,,,,,,,,,,,
|
|
arxiv:2603.11611,Mohammad Aflah Khan,2026,"Fractional Rotation, Full Potential? Investigating Performance and Convergence of Partial RoPE",arXiv,maybe [IC] NoPE loss spikes mitigated by QK-norm (side finding),,,,,,,,,,,,,
|
|
arxiv:2608.30320,Zi-Han Qiu,2026,"On the Design of Qwen3.8-Next Architecture: Evaluation, Efficiency, and Training Stability",,maybe [IC] industry architecture study incl. training stability,,,,,,,,,,,,,
|
|
arxiv:2609.14715,Dushyant Rajput,2026,Depth and Scale in the Sub-150M Regime: JugnuLM-53M vs JugnuLM-110M,arXiv,"maybe [IS] sub-150M depth study (QK-norm fixed, not varied)",,,,,,,,,,,,,
|
|
doi:10.5281/zenodo.18924690,Hang Lu,2026,Modernizing GPT-2-Scale Models Under a 10B-Token Training Budget,Zenodo (CERN European Organization for Nuclear Research),"maybe [IS] GPT-2-scale modernisation incl. QK-norm; Zenodo, check quality",,,,,,,,,,,,,
|
|
arxiv:2507.13338,Laker Newhouse,2025,Training Transformers with Enforced Lipschitz Constants,arXiv,maybe [IA] Lipschitz transformers trained without QK-norm/softcap,,,,,,,,,,,,,
|
|
arxiv:2505.22014,Jörg K. H. Franke,2025,Learning in Compact Spaces with Approximately Normalized Transformer,arXiv,maybe [IA] approx. normalised transformer vs GPT+QK-norm convergence,,,,,,,,,,,,,
|
|
arxiv:2503.10622,Jia-Cheng Zhu,2025,Transformers without Normalization,Computer Vision and Pattern Recognition,maybe [IA] DyT replaces norm layers,,,,,,,,,,,,,
|
|
arxiv:2601.02031,Felix Stollenwerk,2026,Output Embedding Centering for Stable LLM Pretraining,arXiv.org,maybe [IA] output-logit divergence; z-loss / softcap context,,,,,,,,,,,,,
|
|
arxiv:2309.17453,Guang-Xuan Xiao,2023,Efficient Streaming Language Models with Attention Sinks,International Conference on Learning Representations,"maybe [IM] attention sinks (inference paper, mechanism background)",,,,,,,,,,,,,
|
|
arxiv:2410.13835,Tian-Yu Guo,2024,Active-Dormant Attention Heads: Mechanistically Demystifying Extreme-Token Phenomena in LLMs,arXiv.org,maybe [IM] extreme-token phenomena mechanism,,,,,,,,,,,,,
|
|
arxiv:2501.12352,Ke Alexander Wang,2025,Test-time regression: a unifying framework for designing sequence models with associative memory,arXiv,maybe [IM] theory justifying QK-norm (test-time regression view),,,,,,,,,,,,,
|
|
arxiv:2502.00919,Stephen Zhang,2025,"Attention Sinks: A 'Catch, Tag, Release' Mechanism for Embeddings",arXiv,maybe [IM] attention sinks persist with QK-norm,,,,,,,,,,,,,
|
|
arxiv:2510.22026,Nikita Karagodin,2025,Normalization in Attention Dynamics,Neural Information Processing Systems,maybe [IM] normalisation in attention dynamics,,,,,,,,,,,,,
|
|
arxiv:2608.02091,Shuxiao Xie,2026,"One QK Channel, Many Sources: Tracing Low-Precision Attention Collapse",arXiv,maybe [IM] low-precision attention collapse; QK-norm disrupts it,,,,,,,,,,,,,
|
|
arxiv:2609.34272,Jun-Lin Chen,2026,Broken Symmetry in BF16 Attention: Why FlashAttention Gradients Blow Up Late in Training,,maybe [IM] BF16 attention gradients; logit growth late in training,,,,,,,,,,,,,
|
|
arxiv:2606.28116,Rui-Xuan Huang,2026,Mechanism-Aligned Observability for Training Instability of Large Language Models,,maybe [IM] instability diagnostics; possible metrics,,,,,,,,,,,,,
|
|
arxiv:2601.22966,Zihan Qiu,2026,A Unified View of Attention and Residual Sinks: Outlier-Driven Rescaling is Essential for Transformer Training,arXiv (Cornell University),"maybe [IM] attention + residual sinks, outlier rescaling",,,,,,,,,,,,,
|
|
arxiv:2602.18849,S. Emadi,2026,Exact Attention Sensitivity and the Geometry of Transformer Stability,arXiv.org,maybe [IM] attention sensitivity and stability geometry,,,,,,,,,,,,,
|
|
arxiv:2304.03208,Nolan Dey,2023,Cerebras-GPT: Open Compute-Optimal Language Models Trained on the Cerebras Wafer-Scale Cluster,arXiv.org,maybe [BG] muP in small open models,,,,,,,,,,,,,
|
|
arxiv:2407.07972,Rosie Zhao,2024,Deconstructing What Makes a Good Optimizer for Language Models,International Conference on Learning Representations,"maybe [BG] LR stability across optimisers, small LMs",,,,,,,,,,,,,
|
|
arxiv:2405.13698,Xi Wang,2024,How to set AdamW's weight decay as you scale model and dataset size,International Conference on Machine Learning,maybe [BG] AdamW hyperparameters vs scale,,,,,,,,,,,,,
|
|
arxiv:2510.19093,Atli Kosson,2025,Weight Decay may matter more than muP for Learning Rate Transfer in Practice,arXiv.org,maybe [BG] LR transfer without muP,,,,,,,,,,,,,
|
|
arxiv:2505.01618,Nolan Dey,2025,Don't be lazy: CompleteP enables compute-efficient deep transformers,Neural Information Processing Systems,"maybe [BG] depth-wise parametrisation, LR across depth",,,,,,,,,,,,,
|
|
arxiv:2608.24814,Zi-Han Liu,2026,Effective Learning Rate Governs Loss Dynamics in Language Model Pretraining,,maybe [BG] effective LR view of loss dynamics,,,,,,,,,,,,,
|