Organise MSc thesis project into one folder
This commit is contained in:
Cormac Shannon committed
commit
45f46ba3cf
17 files changed
+8059
No files matched your search
File diff suppressed because it is too large.
Load diff
@@ -0,0 +1,59 @@
|
||||
# CS6513 Paper — Introduction + Mini Literature Review
|
||||
|
||||
IEEE conference-format manuscript for CS6513. Source: `main.tex`.
|
||||
|
||||
## Build
|
||||
|
||||
```sh
|
||||
./paper/build.sh # run from the repo root
|
||||
```
|
||||
|
||||
`build.sh` auto-detects a TeX engine (and picks up MacTeX / TeX Live even if it
|
||||
isn't on your `PATH`). It tries, in order: `latexmk` → `pdflatex` (two passes) →
|
||||
`tectonic`. Output: `paper/main.pdf`.
|
||||
|
||||
### Toolchain
|
||||
|
||||
No system TeX install is needed if you use **tectonic** (what this repo was
|
||||
built with):
|
||||
|
||||
```sh
|
||||
brew install tectonic
|
||||
```
|
||||
|
||||
Tectonic is a single self-contained binary (~20 MB) that downloads the exact TeX
|
||||
packages the document needs from CTAN on first run and caches them.
|
||||
|
||||
Alternatives:
|
||||
- **MacTeX** (full TeX Live, ~4 GB): `brew install --cask mactex-no-gui`
|
||||
- **BasicTeX** (small, ~100 MB; may need `sudo tlmgr install ...` for extras)
|
||||
- **Overleaf** (zero install): upload/clone this `paper/` folder and compile `main.tex`.
|
||||
|
||||
## Required LaTeX packages
|
||||
|
||||
All standard and bundled with a normal TeX distribution (and auto-fetched by
|
||||
tectonic):
|
||||
|
||||
| Package | Why |
|
||||
|---|---|
|
||||
| `IEEEtran` (document class) | IEEE conference format — `IEEEtran.cls` is committed here so the repo is self-contained |
|
||||
| `cite` | citation handling |
|
||||
| `amsmath`, `amssymb`, `amsfonts` | math symbols (μ, ×, →, etc.) |
|
||||
| `graphicx` | figures (none yet) |
|
||||
| `textcomp` | text symbols |
|
||||
| `xcolor` | colour |
|
||||
| `url` | URLs in the bibliography |
|
||||
|
||||
## Notes learned while building
|
||||
|
||||
- **The first tectonic run needs a network connection** — it fetches packages
|
||||
from CTAN. Later builds use the cache and are fast/offline.
|
||||
- **Transient `HTTP 429 Too Many Requests`** from tectonic's package server can
|
||||
interrupt that first build. It's not a document error — **just re-run
|
||||
`build.sh`**; tectonic resumes from what it already cached.
|
||||
- **`Underfull \hbox (badness ...)`** warnings are cosmetic — short lines (e.g.
|
||||
bullet lists) that don't fill the justified column. Harmless, and they go away
|
||||
as the draft becomes prose. There are currently no real errors.
|
||||
- Build artifacts (`*.aux`, `*.log`, …) and the compiled `main.pdf` are
|
||||
git-ignored. Commit the **final** submission PDF explicitly (`git add -f`) if
|
||||
you want it tracked.
|
||||
Executable
+50
@@ -0,0 +1,50 @@
|
||||
#!/usr/bin/env bash
|
||||
# Build a CS6513 IEEE .tex file (-> matching .pdf).
|
||||
# Usage:
|
||||
# ./paper/build.sh # builds main.tex (the submission)
|
||||
# ./paper/build.sh ref_draft # builds ref_draft.tex (personal reference)
|
||||
# ./paper/build.sh main.tex # .tex extension optional
|
||||
set -euo pipefail
|
||||
cd "$(dirname "$0")"
|
||||
|
||||
# Which document to build (default: main). Accept an optional first arg, with or
|
||||
# without a trailing ".tex".
|
||||
MAIN="${1:-main}"
|
||||
MAIN="${MAIN%.tex}"
|
||||
|
||||
if [ ! -f "$MAIN.tex" ]; then
|
||||
echo "No such file: $MAIN.tex (in $(pwd))" >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
# Pick up a TeX toolchain that may be installed but not on PATH (MacTeX / TeX Live).
|
||||
for d in /Library/TeX/texbin /usr/local/texlive/*/bin/* /opt/homebrew/bin; do
|
||||
if [ -d "$d" ]; then PATH="$d:$PATH"; fi
|
||||
done
|
||||
export PATH
|
||||
|
||||
if command -v latexmk >/dev/null 2>&1; then
|
||||
echo "Building with latexmk..."
|
||||
latexmk -pdf -interaction=nonstopmode -halt-on-error "$MAIN.tex"
|
||||
elif command -v pdflatex >/dev/null 2>&1; then
|
||||
echo "Building with pdflatex (two passes to resolve \\cite references)..."
|
||||
pdflatex -interaction=nonstopmode -halt-on-error "$MAIN.tex"
|
||||
pdflatex -interaction=nonstopmode -halt-on-error "$MAIN.tex"
|
||||
elif command -v tectonic >/dev/null 2>&1; then
|
||||
echo "Building with tectonic..."
|
||||
tectonic "$MAIN.tex"
|
||||
else
|
||||
cat >&2 <<'EOF'
|
||||
No LaTeX engine found (looked for latexmk, pdflatex, tectonic).
|
||||
Install one, then re-run this script:
|
||||
Tectonic (lightest, self-contained ~30MB, fetches packages on demand):
|
||||
brew install tectonic
|
||||
BasicTeX (small TeX Live ~100MB; may need `sudo tlmgr install` for extras):
|
||||
brew install --cask basictex
|
||||
MacTeX (full TeX Live ~4GB, everything included):
|
||||
brew install --cask mactex-no-gui
|
||||
EOF
|
||||
exit 1
|
||||
fi
|
||||
|
||||
echo "OK -> $(pwd)/$MAIN.pdf"
|
||||
@@ -0,0 +1,30 @@
|
||||
# Abstract.
|
||||
|
||||
Modern language-model architectures ship many components which have been validated using large models, however, whether the same benefits are reproduced for smaller models is actively disputed. This paper aims to produce a controlled, isolated study of one such component: Query-Key Normalisation (QK-norm) against sub 100M parameter models, defining when it becomes necessary (at which learning rate and model depths).
|
||||
|
||||
# Index Terms.
|
||||
|
||||
small language models
|
||||
transformer architecture
|
||||
training stability
|
||||
query–key normalization
|
||||
learning rate
|
||||
ablation study
|
||||
|
||||
# Introduction.
|
||||
|
||||
The transformer architecture has quickly become a domainant choice in the natural language processing (NLP) and machine learning (ML) field. While popularly used in large scale models boasting hundreds of billions to trillions of parameters, the transformer architecture has also seen growing success in smaller models, running inference on edge-compute. These models are cheaper to run and store on low-end hardware, efficient models can run with only a few tens of millions of parameters.
|
||||
Modern LLM Architecture are the culmination of many components, including but not limited to: rotary position embeddings, RMSNorm, grouped-query attention, SwiGLU, and QK-norm. ALmost all of which have been validated against models billions of parameters in size [2], [5]. Although we know these techniques provided benefit in large models, it is not clearly established that these components carry their benefits to small models. Guidance for composing sub-100M parameter models is limited, and for several of the components mentioned above, the evidence from literature is contradictory regarding their benefits. Gaining clarity in this area is important, small-model pretraining is a active area of research (sample-efficient work[6], compute-constrained ablations [7]), choosing the right components matters when compute budget is tightest.
|
||||
|
||||
QK-Norm is a modifcation to a transformers self-attention step. First proposed by Henry et al. [1] for use in low-resource neural machine translation [1]. The technique scales both the query and key vectors to a fixed magnitude before their dot product is calculated to create their attention logits. Without this step, the attention logits can grow without bound after training for many steps, causing the proceeding softmax step to create a near one-hot ourcome. The learning gradients become vanishingly small, and the training loss can start spiking or diverge, requiring the learning rate to be reduced [3], [4]. Recent mechanical studies associate the reason behind logit growth to "massive-activations", wherein a small number of tokens gain extreme-mangitude activations and become the domainant influence over the key vector, forming "attention sinks" [10], [11]. QK-Norm resolves this issue by preventing unbound growth of these vectors, thus removing the failure mode.
|
||||
QK-Norm was popularised by the 22-billion-parameter Vision Transformer (ViT-22B) of Dehghani et al. [2], the technique has since grown to become a near standard step in frontier language models. For example, OLMo 2 utilised RMSNorm to the Q and K vectors and found improvements to their models stability and training loss [5]. The technique has since been adopted by Gemma 2/3, Qwen3, and Chameleon.
|
||||
However, QK-norm stands among several logit-control techniques: Gemma 2 soft-capped their attention logits using a smooth saturating function [12], Z-loss maintains softmax normalisers near zero preventing drifting [13], and NormFormer adds an extra normalisation layer inside each block [14]. Each of these techniques are frequently studied together, so the impact of just QK-Norm's is difficult to observe.
|
||||
Literature regarding small models is sparse, and of what exists, often entangled. Wortsman et al. provides a solid methodological template. Wortsman found that instabilities found in large models can be reproduced in small models at high learning rates, QK-layernorm paired with z-loss can resolve these issues. They found this behaviour transfers across models sizes. They also introduce a learning-rate (LR) sensitivity metric that measures how far a model's final loss deviates from its optimum as the learning rate is adjusted. Recently, IMU-1 by Grigorev et al. found in 70M parameter models, the single most postively impactful architecture change was QK-Norm. This was measured via reducing the activiation kurtosis [6]. However, in their study, they paired QK-Norm alongside maximal update parametrisation (µP), NorMuon, and value residuals.
|
||||
It is important ot note that these papers do not stand without objection. Other literature finds that those sames ablations (RMSNorm, rotary position embeddings, grouped-query attention) provide no measurable benefit to models sized at 70M parameters [7]. The two literatures disagree at effectively the same scale, while neither paper conducts the isolated learning-rate experiment that would settle the matter.
|
||||
From this, learning rate arises as the right axis to work along. Maximal-update parametrisation details that optimal learning rate can transfer across model widths [8]. Learning rate is the hyperparameter that these stability effects are dependant on. Therefore, a faithful examination of QK-Norm requires us to study it as it relates to learning rate while maximal-update parametrisation is disabled, ensuring that its effects do not mask the behaviour we are studying.
|
||||
In regards to evidence inspecting the behaviour of small models, there are two limitations surounding them. First, evidence studying the impact on small models is confounded, Wortsman pairs qk-norm with z-loss and uses small models as a proxy for larger models, while IMU-1 pairs it with several other techniques (µP + NorMuon + value residuals) measuring its effectiveness through activation kurtosis at a single fixed depth and never sweeping learning rate [6]. Second, the opposing works test against a compute-optimal budget and does not isolate QK-norm or sweep learning as we desire [7]. The arising issue is that there is no study isolating the behaviour of QK-norm through different learning rates or at what model depth it bcomes necessary. A second gap revolves around accessibility, most texts harnesses expensive GPUs, making reproducability difficult. A laptop-reproducable study on its own is a contribution on its own.
|
||||
This study therefore aims to isolate the effect of QK-norm and to determine at what scale a small language model benefits from it. It ask, at fewer than 100M parameters, does QK-norm - on its own - improve model convergense and raise the maximum stable learning rate? Additionally, at what learning rate does it become necessary? As a final subquestion, how does that threshold shift as the model depth is changed?
|
||||
Three hypothesis can there be derived: First, the effect of QK-norm is learning-rate conditional, negligible impact or marginally harmful at low learning rates but highly impactful at higher ones (H1); that QK-Norm raises the max stable learning rate and lowers learning-rate-sensitivity (H2); and that the learning rate at which QK-norm becomes necessary lowers as depth increases.
|
||||
The study will investigate these hypothesis by conducting controlled ablations/changes on a nanoGPT [9] project with maximal update parametrisation disabled, AdamW set as the optimiser, while toggling QK-norm as the sole independant variable through a dense sweep of learning rates, and additionally against a small range of model depths at a model scale of roughly 10-15M parameters.
|
||||
Results will be measured through the learning-rate sensitivity [4] , the maximum stable learning rate, steps-to-threshold, perplexity, attention entropy and activation kurtosis.
|
||||
The project will be working with small models, which has a supported precident to produce cohesive text by using the TinyStories [16] body of text, which can train on a single-GPU as establish in the Cramming studying [17]. The project is designed to run on a consumer laptop (Apple M4 Pro chip), with code, seeds, and logs distributed.
|
||||
Binary file not shown.
@@ -0,0 +1,235 @@
|
||||
%% =====================================================================
|
||||
%% draft.tex -- LaTeX conversion of draft.md (the author's own prose)
|
||||
%% =====================================================================
|
||||
%% This is a FORMAT conversion only: the Introduction/Abstract prose is
|
||||
%% reproduced VERBATIM from draft.md (Cormac's own words, typos included --
|
||||
%% nothing here is copy-edited). Errors were REPORTED back in chat rather than
|
||||
%% silently fixed, so the repo keeps showing your own incremental writing (the
|
||||
%% point of the anti-AI audit). Only structure was added: markdown headings ->
|
||||
%% IEEE sections, and the numeric [1]..[17] citations -> \cite{} keys.
|
||||
%%
|
||||
%% CITATIONS FILLED (draft.md named the source but gave no [n]) -- both flagged
|
||||
%% inline with "% [FILLED CITATION ...]" so you can confirm/adjust:
|
||||
%% * "Wortsman ... methodological template" -> \cite{wortsman2024}
|
||||
%% * Gap para, "Wortsman pairs qk-norm with z-loss" -> \cite{wortsman2024}
|
||||
%%
|
||||
%% Number mapping (key -> printed number). qknmla (was [15] in main.tex) is
|
||||
%% never cited in this draft, so it was REMOVED; LaTeX renumbers the rest from
|
||||
%% \bibitem order, so tinystories/cramming are now [15]/[16] (were [16]/[17]).
|
||||
%% NB: this draft's numbering therefore no longer matches main.tex's 17-item list.
|
||||
%% [1]=henry2020 [2]=vit22b [3]=zhai2023 [4]=wortsman2024 [5]=olmo2
|
||||
%% [6]=imu1 [7]=slmtradeoffs [8]=mup [9]=nanogpt [10]=sun_massive
|
||||
%% [11]=attnsink [12]=softcap [13]=zloss [14]=normformer [15]=tinystories
|
||||
%% [16]=cramming
|
||||
%%
|
||||
%% Title/author block carried over from main.tex (draft.md had none).
|
||||
%% Build: ./paper/build.sh draft
|
||||
%% =====================================================================
|
||||
|
||||
\documentclass[conference]{IEEEtran}
|
||||
\IEEEoverridecommandlockouts
|
||||
\usepackage{cite}
|
||||
\usepackage{amsmath,amssymb,amsfonts}
|
||||
\usepackage{graphicx}
|
||||
\usepackage{textcomp}
|
||||
\usepackage{xcolor}
|
||||
\usepackage{url}
|
||||
|
||||
\begin{document}
|
||||
|
||||
% Title/author carried over from main.tex (draft.md did not include them).
|
||||
\title{When Does a Small Language Model Need QK-Norm?\\
|
||||
An Isolated Learning-Rate Characterization of Query--Key Normalization\\
|
||||
at Sub-100M Parameters\\
|
||||
% {\footnotesize \textsuperscript{}(working title -- draft)}
|
||||
}
|
||||
|
||||
\author{\IEEEauthorblockN{Cormac Shannon}
|
||||
\IEEEauthorblockA{\textit{MSc Immersive Software Engineering} \\
|
||||
\textit{University of Limerick}\\
|
||||
Limerick, Ireland \\
|
||||
23366303}
|
||||
}
|
||||
|
||||
\maketitle
|
||||
|
||||
\begin{abstract}
|
||||
Modern language-model architectures ship many components which have been
|
||||
validated using large models, however, whether the same benefits are reproduced
|
||||
for smaller models is actively disputed. This paper aims to produce a controlled,
|
||||
isolated study of one such component: Query-Key Normalisation (QK-norm) against
|
||||
sub 100M parameter models, defining when it becomes necessary (at which learning
|
||||
rate and model depths).
|
||||
\end{abstract}
|
||||
|
||||
\begin{IEEEkeywords}
|
||||
small language models, transformer architecture, training stability,
|
||||
query--key normalization, learning rate, ablation study
|
||||
\end{IEEEkeywords}
|
||||
|
||||
\section{Introduction}
|
||||
|
||||
% [Context & Hook]
|
||||
The transformer architecture has quickly become a dominant choice in the natural
|
||||
language processing (NLP) and machine learning (ML) field. While popularly used in
|
||||
large scale models boasting hundreds of billions to trillions of parameters, the
|
||||
transformer architecture has also seen growing success in smaller models, running
|
||||
inference on edge-compute. These models are cheaper to run and store on low-end
|
||||
hardware, efficient models can run with only a few tens of millions of parameters.
|
||||
|
||||
Modern LLM Architecture are the culmination of many components, including but not
|
||||
limited to: rotary position embeddings, RMSNorm, grouped-query attention, SwiGLU,
|
||||
and QK-norm. Almost all of which have been validated against models billions of
|
||||
parameters in size \cite{vit22b, olmo2}. Although we know these techniques provided
|
||||
benefit in large models, it is not clearly established that these components carry
|
||||
their benefits to small models. Guidance for composing sub-100M parameter models is
|
||||
limited, and for several of the components mentioned above, the evidence from
|
||||
literature is contradictory regarding their benefits. Gaining clarity in this area
|
||||
is important, small-model pretraining is an active area of research (sample-efficient
|
||||
work \cite{imu1}, compute-constrained ablations \cite{slmtradeoffs}), choosing the
|
||||
right components matters when compute budget is tightest.
|
||||
|
||||
% [Literature review]
|
||||
QK-Norm is a modification to a transformers self-attention step. First proposed by
|
||||
Henry et al. \cite{henry2020} for use in low-resource neural machine translation.
|
||||
The technique scales both the query and key vectors to a fixed
|
||||
magnitude before their dot product is calculated to create their attention logits.
|
||||
Without this step, the attention logits can grow without bound after training for
|
||||
many steps, causing the subsequent softmax step to create a near one-hot outcome.
|
||||
The learning gradients become vanishingly small, and the training loss can start
|
||||
spiking or diverge, requiring the learning rate to be reduced \cite{zhai2023,
|
||||
wortsman2024}. Recent mechanical studies associate the reason behind logit growth to
|
||||
``massive-activations'', wherein a small number of tokens gain extreme-magnitude
|
||||
activations and become the domainant influence over the key vector, forming
|
||||
``attention sinks'' \cite{sun_massive, attnsink}. QK-Norm resolves this issue by
|
||||
preventing unbound growth of these vectors, thus removing the failure mode.
|
||||
|
||||
QK-Norm was popularised by the 22-billion-parameter Vision Transformer (ViT-22B) of
|
||||
Dehghani et al. \cite{vit22b}, the technique has since grown to become a near
|
||||
standard step in frontier language models. For example, OLMo 2 utilised RMSNorm to
|
||||
the Q and K vectors and found improvements to their models stability and training
|
||||
loss \cite{olmo2}. The technique has since been adopted by Gemma 2/3, Qwen3, and
|
||||
Chameleon.
|
||||
|
||||
However, QK-norm stands among several logit-control techniques: Gemma 2 soft-capped
|
||||
their attention logits using a smooth saturating function \cite{softcap}, Z-loss
|
||||
maintains softmax normalisers near zero preventing drifting \cite{zloss}, and
|
||||
NormFormer adds an extra normalisation layer inside each block \cite{normformer}.
|
||||
Each of these techniques are frequently studied together, so the impact of just
|
||||
QK-Norm alone is difficult to observe.
|
||||
|
||||
Literature regarding small models is sparse, and of what exists, often entangled.
|
||||
Wortsman et al. provides a solid methodological template \cite{wortsman2024}. % [FILLED CITATION -- draft.md named Wortsman but gave no [n] here]
|
||||
Wortsman found that instabilities found in large models can be reproduced in small
|
||||
models at high learning rates, QK-layernorm paired with z-loss can resolve these
|
||||
issues. They found this behaviour transfers across model sizes. They also introduce
|
||||
a learning-rate (LR) sensitivity metric that measures how far a model's final loss
|
||||
deviates from its optimum as the learning rate is adjusted. Recently, IMU-1 by
|
||||
Grigorev et al. found in 70M parameter models, the single most positively impactful
|
||||
architecture change was QK-Norm. This was measured via reducing the activation
|
||||
kurtosis \cite{imu1}. However, in their study, they paired QK-Norm alongside maximal
|
||||
update parametrisation ($\mu$P), NorMuon, and value residuals.
|
||||
|
||||
It is important to note that these papers do not stand without objection. Other
|
||||
literature finds that some of the same ablations (RMSNorm, rotary position embeddings,
|
||||
grouped-query attention) do not transfer well to models sized at 70M parameters
|
||||
\cite{slmtradeoffs}. The two literatures disagree at effectively the same
|
||||
scale, while neither paper conducts the isolated learning-rate experiment that would
|
||||
settle the matter.
|
||||
|
||||
From this, learning rate arises as the right axis to work along. Maximal-update
|
||||
parametrisation details that optimal learning rate can transfer across model widths
|
||||
\cite{mup}. Learning rate is the hyperparameter that these stability effects are
|
||||
dependant on. Therefore, a faithful examination of QK-Norm requires us to study it
|
||||
as it relates to learning rate while maximal-update parametrisation is disabled,
|
||||
ensuring that its effects do not mask the behaviour we are studying.
|
||||
|
||||
% [Problem statement & gap]
|
||||
In regards to evidence inspecting the behaviour of small models, there are two
|
||||
limitations surrounding them. First, evidence studying the impact on small models is
|
||||
confounded, Wortsman pairs qk-norm with z-loss and uses small models as a proxy for
|
||||
larger models \cite{wortsman2024}, while IMU-1 pairs it with several other techniques % [FILLED CITATION -- draft.md named Wortsman but gave no [n] here]
|
||||
($\mu$P + NorMuon + value residuals) measuring its effectiveness through activation
|
||||
kurtosis at a single fixed depth and never sweeping learning rate \cite{imu1}.
|
||||
Second, the opposing works test against a compute-optimal budget and does not isolate
|
||||
QK-norm or sweep learning as we desire \cite{slmtradeoffs}. The arising issue is that
|
||||
there is no study isolating the behaviour of QK-norm through different learning rates
|
||||
or at what model depth it becomes necessary. A second gap revolves around
|
||||
accessibility, most texts harness expensive GPUs, making reproducibility difficult.
|
||||
A laptop-reproducible study on its own is a worthwhile contribution.
|
||||
|
||||
% [Proposed contribution: objective + research questions]
|
||||
This study therefore aims to isolate the effect of QK-norm and to determine at what
|
||||
scale a small language model benefits from it. It asks, at fewer than 100M parameters,
|
||||
does QK-norm --- on its own --- improve model convergence and raise the maximum
|
||||
stable learning rate? Additionally, at what learning rate does it become necessary?
|
||||
As a final subquestion, how does that threshold shift as the model depth is changed?
|
||||
|
||||
% [Hypotheses]
|
||||
Three hypotheses can there be derived: First, the effect of QK-norm is learning-rate
|
||||
conditional, negligible impact or marginally harmful at low learning rates but highly
|
||||
impactful at higher ones (H1); that QK-Norm raises the max stable learning rate and
|
||||
lowers learning-rate-sensitivity (H2); and that the learning rate at which QK-norm
|
||||
becomes necessary lowers as depth increases (H3).
|
||||
|
||||
% [Method]
|
||||
The study will investigate these hypothesis by conducting controlled ablations/changes
|
||||
on a nanoGPT \cite{nanogpt} project with maximal update parametrisation disabled,
|
||||
AdamW set as the optimiser, while toggling QK-norm as the sole independent variable
|
||||
through a dense sweep of learning rates, and additionally against a small range of
|
||||
model depths at a model scale of roughly 10-15M parameters.
|
||||
|
||||
% [Metrics]
|
||||
Results will be measured through the learning-rate sensitivity \cite{wortsman2024},
|
||||
the maximum stable learning rate, steps-to-threshold, perplexity, attention entropy
|
||||
and activation kurtosis.
|
||||
|
||||
% [Data + feasibility]
|
||||
The project will be working with small models, which has a supported precedent to
|
||||
produce cohesive text by using the TinyStories \cite{tinystories} body of text, which
|
||||
can train on a single-GPU as established in the Cramming study \cite{cramming}. The
|
||||
project is designed to run on a consumer laptop (Apple M4 Pro chip), with code, seeds,
|
||||
and logs distributed.
|
||||
|
||||
% =====================================================================
|
||||
% Bibliography carried over from main.tex, minus qknmla (which this draft never
|
||||
% cites) -> 16 entries. LaTeX numbers them by \bibitem order, so removing the
|
||||
% uncited entry renumbers every \cite automatically. ALL entries still need
|
||||
% Scopus / primary-PDF checks -- see the % [TO VERIFY] tags.
|
||||
% =====================================================================
|
||||
\begin{thebibliography}{00}
|
||||
\bibitem{henry2020} A. Henry, P. R. Dahl, et al., ``Query-key normalization for
|
||||
transformers,'' in \textit{Findings of EMNLP}, 2020. arXiv:2010.04245. % [TO VERIFY authors]
|
||||
\bibitem{vit22b} M. Dehghani et al., ``Scaling vision transformers to 22 billion
|
||||
parameters,'' 2023. arXiv:2302.05442.
|
||||
\bibitem{zhai2023} S. Zhai et al., ``Stabilizing transformer training by
|
||||
preventing attention entropy collapse,'' in \textit{ICML}, 2023. arXiv:2303.06296.
|
||||
\bibitem{wortsman2024} M. Wortsman et al., ``Small-scale proxies for large-scale
|
||||
transformer training instabilities,'' in \textit{ICLR (oral)}, 2024.
|
||||
arXiv:2309.14322. % [TO VERIFY full author list]
|
||||
\bibitem{olmo2} OLMo~2 Team, ``2 OLMo 2 furious,'' 2025. arXiv:2501.00656.
|
||||
% [TO VERIFY preferred citation format / authors]
|
||||
\bibitem{imu1} G. Grigorev et al., ``IMU-1: Sample-efficient pre-training of
|
||||
small language models,'' 2026. arXiv:2602.02522. % [TO VERIFY authors; preprint]
|
||||
\bibitem{slmtradeoffs} ``Architectural trade-offs in small language models under
|
||||
compute constraints,'' 2025. arXiv:2512.20877. % [TO VERIFY title + authors]
|
||||
\bibitem{mup} G. Yang et al., ``Tensor programs V: Tuning large neural networks
|
||||
via zero-shot hyperparameter transfer,'' 2022. arXiv:2203.03466.
|
||||
\bibitem{nanogpt} A. Karpathy, ``nanoGPT,'' GitHub repository, 2022--.
|
||||
\url{https://github.com/karpathy/nanoGPT}. % software; check IEEE format for repos
|
||||
\bibitem{sun_massive} M. Sun et al., ``Massive activations in large language
|
||||
models,'' in \textit{ICML}, 2024. arXiv:2402.17762. % [TO VERIFY]
|
||||
\bibitem{attnsink} ``When attention sink emerges in language models: an empirical
|
||||
view,'' 2024. arXiv:2410.10781. % [TO VERIFY authors]
|
||||
\bibitem{softcap} Gemma Team, ``Gemma 2,'' technical report, 2024. % [TO VERIFY] arXiv ID + exact soft-capping citation
|
||||
\bibitem{zloss} A. Chowdhery et al., ``PaLM: Scaling language modeling with
|
||||
pathways,'' 2022. % [TO VERIFY] z-loss source; confirm arXiv ID
|
||||
\bibitem{normformer} S. Shleifer et al., ``NormFormer: Improved transformer
|
||||
pretraining with extra normalization,'' 2021. arXiv:2110.09456. % [TO VERIFY authors]
|
||||
\bibitem{tinystories} R. Eldan and Y. Li, ``TinyStories: How small can language
|
||||
models be and still speak coherent English?,'' 2023. arXiv:2305.07759. % [TO VERIFY]
|
||||
\bibitem{cramming} J. Geiping and T. Goldstein, ``Cramming: Training a language
|
||||
model on a single GPU in one day,'' in \textit{ICML}, 2023. arXiv:2212.14034. % [TO VERIFY]
|
||||
\end{thebibliography}
|
||||
|
||||
\end{document}
|
||||
Binary file not shown.
@@ -0,0 +1,513 @@
|
||||
%% =====================================================================
|
||||
%% CS6513 -- Introduction + Mini Literature Review (IEEE conference)
|
||||
%% WORKING DRAFT -- ROUGH PASS (page limit deliberately ignored for now)
|
||||
%%
|
||||
%% STATUS: This is a SCAFFOLD, not final text. The bullets below are
|
||||
%% ideas / papers / arguments to be turned into flowing prose IN YOUR
|
||||
%% OWN WORDS over several commits. Do NOT submit the bullets as-is.
|
||||
%% Remember: declare this AI-assisted outlining in the notebook GenAI log.
|
||||
%%
|
||||
%% Target final structure (2 pages incl. refs), weaving together:
|
||||
%% 1. Context & Hook 2. Literature Context (mini lit review)
|
||||
%% 3. Problem & Gap 4. Proposed Contribution 5. Roadmap
|
||||
%% =====================================================================
|
||||
%%
|
||||
%% ABOUT THESE COMMENTS:
|
||||
%% Anything after a "%" is a LaTeX comment -- ignored by the compiler, so it
|
||||
%% NEVER appears in the PDF. The annotations below are learning notes for the
|
||||
%% author; they are safe to keep, and can be stripped later. Convention used:
|
||||
%% % DEFN: <term> -- plain-language definition of jargon
|
||||
%% % EXPL: ... -- expands on a concept / why it matters
|
||||
%% % NOTE: ... -- drafting instruction to self
|
||||
%% % [TO VERIFY] -- citation detail to confirm against Scopus / the PDF
|
||||
%%
|
||||
%% ---- MINI GLOSSARY (core terms, defined once here) --------------------
|
||||
%% Transformer : the neural-network design behind modern language models
|
||||
%% (Vaswani et al. 2017, "Attention Is All You Need"). Processes all tokens
|
||||
%% in parallel using "self-attention" (below).
|
||||
%% Token : the unit of text a model reads/predicts -- a word or word-piece
|
||||
%% (e.g. "walking" -> "walk" + "ing"). Text is a sequence of tokens.
|
||||
%% Parameters : the model's learned numbers (weights). "Model size" = how many
|
||||
%% there are. 100M = 100 million. Frontier LLMs have 10^9-10^12 (billions+).
|
||||
%% Self-attention : the mechanism that lets each token "look at" other tokens.
|
||||
%% For every token it builds three vectors -- a QUERY, a KEY, and a VALUE.
|
||||
%% Query / Key / Value (Q/K/V) : query = "what am I looking for", key = "what do
|
||||
%% I offer", value = "what I pass on". A token attends to others by comparing
|
||||
%% its query to their keys.
|
||||
%% Attention logit : the raw match score between a query and a key (their dot
|
||||
%% product). Large logits -> the next step (softmax) becomes extreme.
|
||||
%% Softmax : turns a list of scores into probabilities (all positive, sum 1).
|
||||
%% Used to decide how much each token attends to each other token.
|
||||
%% Learning rate (LR) : the step size during training. Too low = slow; too high
|
||||
%% = unstable / diverging. The central variable in THIS study.
|
||||
%% Ablation : an experiment that removes or toggles ONE component to measure
|
||||
%% its specific effect -- the core method here (QK-norm on vs off).
|
||||
%% Perplexity : a standard quality score for language models (= exp of the
|
||||
%% average prediction loss). LOWER is better.
|
||||
%% -----------------------------------------------------------------------
|
||||
|
||||
\documentclass[conference]{IEEEtran}
|
||||
% DEFN: \documentclass[conference]{IEEEtran} -- load the official IEEE conference
|
||||
% style (the IEEEtran.cls file in this folder). Sets the two-column layout,
|
||||
% fonts and margins the assignment requires; do not change.
|
||||
% DEFN: \usepackage{...} -- pulls in an add-on ("package") of extra commands.
|
||||
% Each line below is one such add-on; comments say what it gives us.
|
||||
\IEEEoverridecommandlockouts
|
||||
\usepackage{cite} % tidies up citation numbers, e.g. [2]-[4]
|
||||
\usepackage{amsmath,amssymb,amsfonts} % maths symbols: the arrows, <, mu, etc.
|
||||
\usepackage{graphicx} % lets us include figures/images (none yet)
|
||||
\usepackage{textcomp} % extra text symbols
|
||||
\usepackage{xcolor} % coloured text (if ever needed)
|
||||
\usepackage{url} % formats web links in the bibliography
|
||||
|
||||
\begin{document}
|
||||
% EXPL: everything the reader SEES lives between \begin{document} and
|
||||
% \end{document}. Above that line is the "preamble" (setup only).
|
||||
|
||||
% DEFN: \title{...} sets the paper title. The "\\" forces a line break.
|
||||
% DEFN: QK-Norm (Query-Key Normalization) -- the technique under study: before
|
||||
% attention compares a query to a key, rescale both to a controlled length
|
||||
% so their match score (logit) cannot blow up. Full explanation in Sec. below.
|
||||
% DEFN: "Isolated" -- we change ONLY QK-norm, holding everything else fixed, so any
|
||||
% effect is attributable to it alone (see "ablation" in the glossary).
|
||||
% DEFN: "Sub-100M parameters" -- models with fewer than 100 million weights; small
|
||||
% enough to train on a laptop (frontier models are 100-1000x+ larger).
|
||||
\title{When Does a Small Language Model Need QK-Norm?\\
|
||||
An Isolated Learning-Rate Characterization of Query--Key Normalization\\
|
||||
at Sub-100M Parameters\\
|
||||
{\footnotesize \textsuperscript{}(working title -- rough draft)}}
|
||||
|
||||
\author{\IEEEauthorblockN{Cormac Shannon}
|
||||
\IEEEauthorblockA{\textit{MSc Immersive Software Engineering} \\
|
||||
\textit{University of Limerick}\\
|
||||
Limerick, Ireland \\
|
||||
23366303}
|
||||
}
|
||||
|
||||
\maketitle
|
||||
|
||||
% DEFN: abstract -- the ~150-word summary at the very top of a paper. A reader
|
||||
% decides from this alone whether to read on. Convention: write it LAST.
|
||||
% NOTE(abstract): write LAST, once the argument is settled. Rough placeholder only.
|
||||
\begin{abstract}
|
||||
% ROUGH -- rewrite in own words.
|
||||
% EXPL: "components validated only at large scale" -- modern models are built from
|
||||
% many add-on parts (see the architecture list in Sec. Introduction) that
|
||||
% were tested mainly on huge models; nobody checked they still help small ones.
|
||||
% DEFN: \emph{...} -- emphasis (renders as italics).
|
||||
Modern language-model architectures bundle many components validated only at
|
||||
large scale; whether they help \emph{small} models is under-studied and, for
|
||||
some components, actively disputed. This paper motivates a controlled,
|
||||
isolated study of one such component -- Query--Key Normalization (QK-norm) --
|
||||
at sub-100M-parameter scale, characterizing when (at which learning rates and
|
||||
depths) it becomes necessary. % TODO: 1-2 sentences on method + what we expect.
|
||||
% EXPL: "depth" = number of stacked transformer layers; "necessary" = the point
|
||||
% where turning QK-norm off makes training fail/diverge.
|
||||
\end{abstract}
|
||||
|
||||
% DEFN: keywords -- a handful of index terms so search engines/databases can file
|
||||
% the paper. Pick the phrases a reader would search to find work like yours.
|
||||
\begin{IEEEkeywords}
|
||||
small language models, transformer architecture, training stability,
|
||||
query--key normalization, learning rate, ablation study
|
||||
\end{IEEEkeywords}
|
||||
|
||||
% =====================================================================
|
||||
\section{Introduction}
|
||||
% For a 2-page intro-only paper this whole section IS the paper. Below it is
|
||||
% broken into the 5 required components as comment banners; in the final
|
||||
% version they should read as ONE flowing argument (few/no subheadings).
|
||||
|
||||
%% -------------------------------------------------------------------
|
||||
%% COMPONENT 1 -- CONTEXT & HOOK (broad domain + why relevant NOW)
|
||||
%% -------------------------------------------------------------------
|
||||
% NOTE: 1 short paragraph. Funnel from broad -> the tension that motivates us.
|
||||
% DEFN: NLP = Natural Language Processing (getting computers to work with human
|
||||
% language); ML = Machine Learning. LLM = Large Language Model.
|
||||
% DEFN: "on-device / edge" -- running a model locally on a phone/laptop/sensor
|
||||
% instead of a data-centre; motivates SMALL models (limited memory + power).
|
||||
% EXPL: \textbf{...} = bold. This whole block is bullet notes; final version is
|
||||
% ONE flowing paragraph that funnels broad -> specific.
|
||||
\begin{itemize}
|
||||
\item Broad domain: transformer language models now dominate NLP/ML; but the
|
||||
field is increasingly moving toward \textbf{small / efficient} models
|
||||
(on-device, edge, cost, energy, and \emph{reproducibility}).
|
||||
% DEFN: reproducibility -- whether other researchers can re-run your
|
||||
% experiment and get the same result. A core scientific value, and
|
||||
% easier to guarantee with small models anyone can afford to train.
|
||||
\item Modern LLM architectures are an accretion of components -- rotary
|
||||
position embeddings, RMSNorm, grouped-query attention, SwiGLU,
|
||||
\textbf{QK-norm} -- almost all validated at the billion-parameter scale
|
||||
\cite{olmo2,vit22b}.
|
||||
% EXPL: "accretion of components" = built up by stacking many small
|
||||
% add-ons over the years. The four named ones (context only; NOT
|
||||
% what we study):
|
||||
% DEFN: rotary position embeddings (RoPE) -- how a model encodes WORD
|
||||
% ORDER, by rotating the query/key vectors by an angle set by each
|
||||
% token's position.
|
||||
% DEFN: RMSNorm -- a "normalization" layer that rescales a vector to a
|
||||
% standard size to keep numbers well-behaved during training (a
|
||||
% cheaper variant of LayerNorm).
|
||||
% DEFN: grouped-query attention (GQA) -- a memory-saving trick where
|
||||
% several attention "heads" share one set of keys/values.
|
||||
% DEFN: SwiGLU -- a high-performing choice of activation function (the
|
||||
% little non-linear step inside each layer's feed-forward block).
|
||||
% DEFN: \cite{key} -- inserts a numbered reference; "key" points at a
|
||||
% \bibitem in the bibliography at the bottom of the file.
|
||||
\item Hook / tension: it is \emph{not} established that these components carry
|
||||
down to small models. Design rules for the sub-100M regime are thin,
|
||||
and for several components the evidence is contradictory (see below).
|
||||
% EXPL: THE HOOK. "regime" = a size range with its own behaviour. The
|
||||
% whole paper hangs on this tension: proven-big does not mean
|
||||
% proven-small.
|
||||
\item Why now: small-model pretraining is a live research area
|
||||
(sample-efficiency work \cite{imu1}; compute-constrained ablations
|
||||
\cite{slmtradeoffs}); getting the component choices right matters most
|
||||
exactly where the compute budget is smallest.
|
||||
% DEFN: pretraining -- the big first training phase where a model learns
|
||||
% language from lots of raw text (before any task-specific tuning).
|
||||
% DEFN: sample-efficiency -- getting good results from LESS training data.
|
||||
\end{itemize}
|
||||
|
||||
%% -------------------------------------------------------------------
|
||||
%% COMPONENT 2 -- LITERATURE CONTEXT (mini lit review: what's been done)
|
||||
%% -------------------------------------------------------------------
|
||||
% NOTE: the meat. Synthesize (don't list). Group by theme, not by paper.
|
||||
% EXPL: COMPONENT 2 is the "mini literature review": what is already known.
|
||||
% Group by THEME (what QK-norm is / who uses it / small-scale evidence /
|
||||
% the contradiction / why LR), not one-paragraph-per-paper.
|
||||
\begin{itemize}
|
||||
\item \textbf{What QK-norm is + why it exists.}
|
||||
\begin{itemize}
|
||||
\item Origin: Henry et al. \cite{henry2020} -- normalize the query and key
|
||||
vectors before the attention dot product (for low-resource NMT).
|
||||
% DEFN: "normalize ... before the dot product" -- rescale the query and
|
||||
% key vectors to a fixed length first, so their dot product (the
|
||||
% match score) stays in a controlled range. THAT is QK-norm.
|
||||
% DEFN: dot product -- a single number measuring how aligned two vectors
|
||||
% are; here it is the query-key match score = the attention logit.
|
||||
% DEFN: NMT -- Neural Machine Translation (automatic translation). QK-norm
|
||||
% was first proposed there, on small models -- useful for our story.
|
||||
\item Mechanism/problem it solves: without it, query/key norms grow during
|
||||
training $\rightarrow$ large attention logits $\rightarrow$ softmax
|
||||
saturates (near one-hot) $\rightarrow$ unstable/diverging loss,
|
||||
forcing smaller learning rates.
|
||||
% EXPL: the core causal chain, in words --
|
||||
% 1. "norms grow" = the Q/K vectors get longer (bigger magnitude);
|
||||
% 2. so their dot products (attention logits) get very large;
|
||||
% 3. SOFTMAX SATURATES: given one huge score, softmax puts nearly all
|
||||
% weight on a single token -> output is "one-hot" (a spike);
|
||||
% 4. at a spike the gradient (learning signal) ~vanishes, so training
|
||||
% becomes unstable or DIVERGES (loss shoots up), which forces you
|
||||
% to use a smaller, slower learning rate.
|
||||
% DEFN: one-hot -- a vector that is 1 in a single spot and 0 everywhere
|
||||
% else; here it means attention collapses onto ONE token.
|
||||
% DEFN: diverging loss -- training "blowing up": the error metric
|
||||
% increases uncontrollably instead of decreasing.
|
||||
% EXPL: QK-norm breaks step 1->2, so the rest of the chain never fires.
|
||||
% This is exactly why it should let you train at HIGHER LR (our H2).
|
||||
\item Deeper mechanism (why logits grow): ``massive activations'' -- a few
|
||||
tokens acquire extreme-magnitude activations that become dominant key
|
||||
vectors and form ``attention sinks'' \cite{sun_massive,attnsink};
|
||||
QK-norm bounds Q/K magnitude, limiting this. % [TO VERIFY]
|
||||
% DEFN: activation -- an internal value a neuron/layer outputs as data
|
||||
% flows through the network.
|
||||
% DEFN: massive activations -- a handful of these internal values become
|
||||
% hugely larger than the rest (extreme outliers).
|
||||
% DEFN: attention sink -- a token that soaks up a large share of every
|
||||
% other token's attention regardless of relevance (a side-effect of
|
||||
% those massive activations). This is the mechanism QK-norm curbs.
|
||||
\item Popularized at scale for stability: ViT-22B \cite{vit22b}.
|
||||
% DEFN: ViT-22B -- a 22-billion-parameter Vision Transformer (Google).
|
||||
% Widely cited as the work that made QK-norm mainstream for
|
||||
% stabilizing very large models.
|
||||
\item Complementary mechanistic account: ``attention entropy collapse''
|
||||
-- low attention entropy co-occurs with instability
|
||||
\cite{zhai2023}. (QK-norm bounds logits $\rightarrow$ keeps entropy up.)
|
||||
% DEFN: attention entropy -- a spread-out measure of the attention
|
||||
% weights. HIGH entropy = attention shared across many tokens;
|
||||
% LOW entropy = concentrated on a few (spiky). "Collapse" = it
|
||||
% drops too low, which coincides with unstable training. Same story
|
||||
% as softmax saturation, viewed through an information-theory lens.
|
||||
\end{itemize}
|
||||
\item \textbf{QK-norm is now standard in frontier models.}
|
||||
\begin{itemize}
|
||||
\item OLMo 2 applies RMSNorm to Q and K and reports improved training loss
|
||||
+ stability \cite{olmo2}. Also adopted by Gemma~2/3, Qwen3, Chameleon.
|
||||
% TODO: pick 1-2 to cite precisely; verify each.
|
||||
% DEFN: "frontier models" -- the current top-tier, largest LLMs. Naming
|
||||
% several (OLMo 2, Gemma, Qwen3, Chameleon) shows QK-norm is not
|
||||
% niche -- it is standard practice -> the study is relevant.
|
||||
\end{itemize}
|
||||
\item \textbf{QK-norm is one of several logit-control methods} (matters for the gap).
|
||||
\begin{itemize}
|
||||
\item Alternatives/complements: attention logit soft-capping (Gemma~2,
|
||||
$z\!\leftarrow\!c\tanh(z/c)$) \cite{softcap}; z-loss, which keeps the
|
||||
softmax normalizer near zero \cite{zloss}; NormFormer's extra
|
||||
normalization \cite{normformer}. % [TO VERIFY]
|
||||
% EXPL: these are RIVAL ways to tackle the same "logits blow up" problem:
|
||||
% DEFN: logit soft-capping -- squash every attention logit through
|
||||
% z <- c*tanh(z/c), which smoothly caps it at +/-c so it can never
|
||||
% explode ($ ... $ is inline maths; \tanh is the tanh function).
|
||||
% DEFN: z-loss -- a small extra penalty added to the training objective
|
||||
% that keeps the softmax's normalizing sum near 1 (stops it drifting
|
||||
% and destabilizing training).
|
||||
% DEFN: NormFormer -- a recipe that adds a few extra normalization layers
|
||||
% inside each block to steady training.
|
||||
\item These are almost always studied \emph{bundled together}, so QK-norm's
|
||||
isolated effect is rarely attributed. % [TO VERIFY framing]
|
||||
% EXPL: KEY SETUP FOR THE GAP -- if papers turn on QK-norm AND z-loss AND
|
||||
% soft-capping at once, you cannot tell how much QK-norm ALONE did.
|
||||
% "attributed" = credited to a specific cause. Our isolation fixes this.
|
||||
\end{itemize}
|
||||
\item \textbf{Evidence at / near small scale (the closest work).}
|
||||
\begin{itemize}
|
||||
\item Wortsman et al. \cite{wortsman2024} -- instabilities reproduce in
|
||||
\emph{small} models at high LR; qk-layernorm (+ z-loss) mitigate them
|
||||
and ``transfer down''; introduces the \textbf{LR-sensitivity} metric.
|
||||
KEY methodological template for us.
|
||||
% EXPL: the closest prior work + the method we borrow. "transfer down"
|
||||
% = the same fix that works on big models also works on small ones.
|
||||
% DEFN: qk-layernorm -- Wortsman's name for QK-norm (LayerNorm on Q,K).
|
||||
% DEFN: LR-sensitivity -- their metric for HOW MUCH the final loss worsens
|
||||
% as you vary the learning rate. Low sensitivity = robust to LR
|
||||
% choice = a flatter, more forgiving curve. We reuse this metric.
|
||||
\item IMU-1 (Grigorev et al.) \cite{imu1} -- at 70M, QK-norm is the
|
||||
\emph{single largest} architectural win (measured via reduced
|
||||
activation kurtosis) -- but bundled with $\mu$P, NorMuon, value
|
||||
residuals, etc.
|
||||
% EXPL: strongest recent "QK-norm helps small models" datapoint (70M =
|
||||
% 70-million-parameter model) -- BUT confounded (see the Gap).
|
||||
% DEFN: kurtosis -- a statistic for how "outlier-heavy" a set of numbers
|
||||
% is. High kurtosis = a few extreme values (the massive activations
|
||||
% above). QK-norm LOWERING kurtosis = it is taming those outliers.
|
||||
% DEFN: NorMuon / value residuals -- other training tweaks IMU-1 turned on
|
||||
% at the same time; because they are bundled in, IMU-1 cannot isolate
|
||||
% QK-norm's own contribution.
|
||||
\end{itemize}
|
||||
\item \textbf{The countervailing line (the contradiction).}
|
||||
\begin{itemize}
|
||||
\item Compute-constrained small-model ablations report that modern
|
||||
components (RoPE, RMSNorm, GQA) give ``no measurable benefit at
|
||||
$\sim$70M'' \cite{slmtradeoffs}. Directly disputes the above.
|
||||
% EXPL: THE CONFLICT that motivates the paper: one camp says QK-norm is
|
||||
% the biggest small-model win (IMU-1), another says modern parts do
|
||||
% nothing at ~70M. Both look at the same size -> something is untested.
|
||||
% DEFN: "compute-constrained" -- trained under a fixed, limited compute
|
||||
% budget (so, e.g., only one learning-rate setting was tried).
|
||||
\end{itemize}
|
||||
\item \textbf{Why learning rate is the right axis.}
|
||||
\begin{itemize}
|
||||
\item $\mu$P / $\mu$Transfer \cite{mup}: optimal LR transfers across width;
|
||||
LR is the hyperparameter these stability effects hinge on. Motivates
|
||||
studying QK-norm \emph{as a function of LR}, and switching $\mu$P OFF
|
||||
so LR effects are not masked.
|
||||
% DEFN: $\mu$P (muP, "maximal update parametrization") -- a special way of
|
||||
% scaling a network so the BEST learning rate stays the same as you
|
||||
% grow model width; $\mu$Transfer = tuning LR on a small model then
|
||||
% reusing it on a big one. ($\mu$ = the Greek letter mu.)
|
||||
% EXPL: why we turn $\mu$P OFF -- it deliberately makes results LESS
|
||||
% sensitive to learning rate, which would hide the very LR effect we
|
||||
% want to measure. Off = a clean, unmasked view. (Also a difference
|
||||
% from IMU-1, which left it ON.)
|
||||
% DEFN: hyperparameter -- a training setting you choose by hand (like the
|
||||
% learning rate), as opposed to a weight the model learns itself.
|
||||
\item Recent work keeps reaffirming the LR link -- e.g. QK-Normed MLA reports
|
||||
QK-norm ``permits significant increases in learning rate'' \cite{qknmla}.
|
||||
% [TO VERIFY] -- supports H2
|
||||
% EXPL: a 2026 paper independently restating that QK-norm lets you train at
|
||||
% higher LR -- direct external support for our hypothesis H2 below.
|
||||
\end{itemize}
|
||||
\end{itemize}
|
||||
|
||||
%% -------------------------------------------------------------------
|
||||
%% COMPONENT 3 -- PROBLEM STATEMENT & GAP ANALYSIS
|
||||
%% -------------------------------------------------------------------
|
||||
% NOTE: name the gap crisply; it must fall straight out of Component 2.
|
||||
% EXPL: COMPONENT 3 = the hole in the literature the study fills. Every claim here
|
||||
% should already have been set up in Component 2 (no new surprises).
|
||||
\begin{itemize}
|
||||
\item The positive small-scale evidence is \textbf{confounded}: Wortsman
|
||||
bundles qk-norm with z-loss and frames small models as a \emph{proxy}
|
||||
for large-scale behaviour; IMU-1 bundles it with $\mu$P + NorMuon +
|
||||
value residuals and measures it via activation statistics, not an LR
|
||||
sweep, at a single (fixed) depth.
|
||||
% DEFN: confounded -- when two changes are made at once so you cannot tell
|
||||
% which caused the effect; the classic flaw an ablation is meant to avoid.
|
||||
% DEFN: proxy -- a stand-in. Wortsman used small models only as a cheap
|
||||
% stand-in for big-model behaviour, not as the object of interest.
|
||||
% DEFN: LR sweep -- training the same model repeatedly across a range of
|
||||
% learning rates to map out how the outcome depends on LR. The thing
|
||||
% prior work at this scale did NOT do.
|
||||
\item The negative evidence \cite{slmtradeoffs} tests a \emph{single}
|
||||
compute-optimal budget, does not isolate QK-norm, and does not sweep LR.
|
||||
% DEFN: compute-optimal budget -- the amount of training data/steps that is
|
||||
% "best value" for a given model size (Chinchilla-style rule of thumb).
|
||||
% Testing only this one point can miss effects that appear at other LRs.
|
||||
\item So the two literatures \emph{disagree at the same scale} while neither
|
||||
does the clean experiment.
|
||||
% EXPL: one-line crux -- the disagreement persists because the decisive
|
||||
% experiment (isolate + sweep LR) has not been run.
|
||||
\item \textbf{GAP:} no \emph{dense}, \emph{isolated} ($\mu$P-off)
|
||||
learning-rate-sensitivity characterization of QK-norm at 10--100M, and
|
||||
no map of how the LR at which QK-norm becomes \emph{necessary} moves
|
||||
with model depth.
|
||||
% EXPL: the headline gap, four qualifiers doing the work:
|
||||
% dense = many closely-spaced LR values (a smooth curve, not 3 points);
|
||||
% isolated = only QK-norm toggled, nothing else;
|
||||
% muP-off = the masking factor removed (see above);
|
||||
% at 10-100M = the small-scale band nobody has charted this way.
|
||||
\item Secondary gap: reproducibility -- most evidence is expensive / CUDA-only;
|
||||
a laptop-reproducible study is itself a contribution.
|
||||
% DEFN: CUDA -- NVIDIA's GPU software stack; "CUDA-only" work cannot run on
|
||||
% an Apple laptop, so it is hard for others to reproduce cheaply.
|
||||
\end{itemize}
|
||||
|
||||
%% -------------------------------------------------------------------
|
||||
%% COMPONENT 4 -- PROPOSED CONTRIBUTION (objective, RQs, method)
|
||||
%% -------------------------------------------------------------------
|
||||
% NOTE: be explicit and honest -- this is VERIFICATION + boundary-charting,
|
||||
% not a novel-technique claim. Do not say "first".
|
||||
% EXPL: COMPONENT 4 = what WE do about the gap: objective, questions, hypotheses,
|
||||
% method. "boundary-charting" = mapping exactly WHERE QK-norm starts to
|
||||
% matter (which LR, which depth), rather than claiming a new technique.
|
||||
\begin{itemize}
|
||||
\item Objective: cleanly isolate QK-norm and chart \emph{when a small model
|
||||
needs it}.
|
||||
\item Research question: at $<$100M params, does isolated QK-norm improve
|
||||
convergence and raise the maximum stable LR -- and at what LR does it
|
||||
become necessary? Sub-question: how does that threshold shift with depth?
|
||||
% DEFN: convergence -- training successfully settling to a low, stable loss.
|
||||
% "Faster convergence" = reaching a target loss in fewer steps.
|
||||
% DEFN: max stable LR -- the highest learning rate at which training still
|
||||
% succeeds instead of diverging. A key thing QK-norm should raise.
|
||||
\item Hypotheses (state as such):
|
||||
% DEFN: hypothesis -- a specific, testable prediction stated BEFORE running
|
||||
% the experiment. H1-H3 are what we expect the data to show.
|
||||
\begin{itemize}
|
||||
\item H1 -- effect is LR-conditional: negligible/slightly negative at low
|
||||
LR, decisive at high LR.
|
||||
% EXPL: at gentle LRs nothing diverges, so QK-norm barely helps (or
|
||||
% slightly hurts); at aggressive LRs it is what keeps training alive.
|
||||
\item H2 -- QK-norm raises max stable LR and lowers LR-sensitivity.
|
||||
% EXPL: i.e. it widens the band of usable LRs and flattens the loss-vs-LR
|
||||
% curve (more forgiving to the LR you pick).
|
||||
\item H3 -- the ``necessity'' LR falls as depth increases.
|
||||
% EXPL: prediction that DEEPER models (more layers) destabilize sooner, so
|
||||
% they need QK-norm at lower LRs than shallow ones.
|
||||
\end{itemize}
|
||||
\item High-level method: controlled ablation on \textbf{plain nanoGPT}
|
||||
\cite{nanogpt}, $\mu$P off, AdamW; single independent variable
|
||||
QK-norm\{on,off\} $\times$ dense LR sweep $\times$ depth; $\sim$10--15M
|
||||
params; measures = LR-sensitivity \cite{wortsman2024}, max-stable-LR,
|
||||
steps-to-threshold, perplexity, plus attention entropy / activation
|
||||
kurtosis as mechanistic checks.
|
||||
% DEFN: nanoGPT -- Andrej Karpathy's minimal, readable codebase for training
|
||||
% small GPT models; "plain" = without the extra speed-run tricks that
|
||||
% would confound the experiment. Our experimental substrate.
|
||||
% DEFN: AdamW -- the standard optimizer (the algorithm that updates weights
|
||||
% using gradients); "W" = with decoupled weight decay.
|
||||
% DEFN: independent variable -- the ONE thing we deliberately change
|
||||
% (QK-norm on vs off). "$\times$" here means we cross it with every LR
|
||||
% and every depth -> a grid of runs.
|
||||
% DEFN: steps-to-threshold -- how many training steps to first reach a chosen
|
||||
% loss value; a concrete convergence-speed number.
|
||||
% DEFN: mechanistic checks -- extra measurements (attention entropy, kurtosis)
|
||||
% that reveal WHY an effect happens, not just THAT it happens.
|
||||
\item Data + precedent: small models learn coherent language from tiny corpora
|
||||
(TinyStories) \cite{tinystories}; single-GPU near-BERT training is an
|
||||
established accessible-research precedent (Cramming) \cite{cramming}. % [TO VERIFY]
|
||||
% DEFN: corpus (pl. corpora) -- the body of text a model trains on.
|
||||
% DEFN: TinyStories -- a dataset of very simple stories on which even
|
||||
% sub-10M-param models produce fluent English; justifies our tiny scale.
|
||||
% DEFN: Cramming -- a paper training a BERT-quality model on ONE GPU in ONE
|
||||
% day; the precedent for "serious results on modest hardware".
|
||||
% DEFN: BERT -- an influential 2018 language model; "near-BERT" = almost as
|
||||
% good as BERT.
|
||||
\item Feasibility/repro: runs on a consumer laptop (Apple M4 Pro); code, seeds,
|
||||
logs released.
|
||||
% DEFN: seed -- the number that fixes the random choices in a training run;
|
||||
% sharing seeds lets anyone reproduce the exact run. Running multiple
|
||||
% seeds also shows an effect is real, not a fluke of one random draw.
|
||||
\end{itemize}
|
||||
|
||||
%% -------------------------------------------------------------------
|
||||
%% COMPONENT 5 -- PAPER ROADMAP (standard closing paragraph)
|
||||
%% -------------------------------------------------------------------
|
||||
% NOTE: rewrite to match the final section list once it exists.
|
||||
% EXPL: COMPONENT 5 = the "roadmap" paragraph. A standard academic convention:
|
||||
% one or two sentences telling the reader how the rest of the paper is laid
|
||||
% out. "Section II ... Section III ..." are the numbered parts that follow.
|
||||
% (Roman numerals are how IEEEtran auto-numbers \section headings.)
|
||||
\begin{itemize}
|
||||
\item ``The remainder of this paper is structured as follows: Section II
|
||||
reviews related work on QK-norm and small-model training stability;
|
||||
Section III details the experimental design; Section IV reports results;
|
||||
and Section V concludes.'' % adjust to actual sections.
|
||||
\end{itemize}
|
||||
|
||||
% =====================================================================
|
||||
% NOTE(refs): inline thebibliography for now (matches template). Switch to a
|
||||
% .bib file later if we want. EVERY entry below still needs its exact
|
||||
% authors/venue/pages checked against the primary PDF -- during this project
|
||||
% AI-surfaced citation details were wrong more than once. arXiv IDs are the
|
||||
% most reliable field; treat author lists / venues as [TO VERIFY].
|
||||
% =====================================================================
|
||||
% DEFN: thebibliography -- the reference list. Each \bibitem{key} defines one
|
||||
% reference; \cite{key} in the text above prints its number. The {00} just
|
||||
% tells LaTeX the labels are up to two digits wide.
|
||||
% DEFN: arXiv -- a free repository of (often not-yet-peer-reviewed) research
|
||||
% papers. "arXiv:2010.04245" is a permanent ID: go to arxiv.org/abs/2010.04245.
|
||||
% Reliable for finding the paper; the AUTHOR/VENUE text still needs checking
|
||||
% (hence the [TO VERIFY] tags) -- run each through Scopus before submission.
|
||||
\begin{thebibliography}{00}
|
||||
|
||||
\bibitem{henry2020} A. Henry, P. R. Dahl, et al., ``Query-key normalization for
|
||||
transformers,'' in \textit{Findings of EMNLP}, 2020. arXiv:2010.04245. % [TO VERIFY authors]
|
||||
|
||||
\bibitem{vit22b} M. Dehghani et al., ``Scaling vision transformers to 22 billion
|
||||
parameters,'' 2023. arXiv:2302.05442.
|
||||
|
||||
\bibitem{zhai2023} S. Zhai et al., ``Stabilizing transformer training by
|
||||
preventing attention entropy collapse,'' in \textit{ICML}, 2023.
|
||||
arXiv:2303.06296.
|
||||
|
||||
\bibitem{wortsman2024} M. Wortsman et al., ``Small-scale proxies for large-scale
|
||||
transformer training instabilities,'' in \textit{ICLR (oral)}, 2024.
|
||||
arXiv:2309.14322. % [TO VERIFY full author list]
|
||||
|
||||
\bibitem{olmo2} OLMo~2 Team, ``2 OLMo 2 furious,'' 2025. arXiv:2501.00656.
|
||||
% [TO VERIFY preferred citation format / authors]
|
||||
|
||||
\bibitem{imu1} G. Grigorev et al., ``IMU-1: Sample-efficient pre-training of
|
||||
small language models,'' 2026. arXiv:2602.02522. % [TO VERIFY authors; preprint]
|
||||
|
||||
\bibitem{slmtradeoffs} ``Architectural trade-offs in small language models under
|
||||
compute constraints,'' 2025. arXiv:2512.20877. % [TO VERIFY title + authors]
|
||||
|
||||
\bibitem{mup} G. Yang et al., ``Tensor programs V: Tuning large neural networks
|
||||
via zero-shot hyperparameter transfer,'' 2022. arXiv:2203.03466.
|
||||
|
||||
\bibitem{nanogpt} A. Karpathy, ``nanoGPT,'' GitHub repository, 2022--.
|
||||
\url{https://github.com/karpathy/nanoGPT}. % software; check IEEE format for repos
|
||||
|
||||
% --- added 18/08/2026: surrounding literature; ALL need Scopus/PDF checks ---
|
||||
\bibitem{sun_massive} M. Sun et al., ``Massive activations in large language
|
||||
models,'' in \textit{ICML}, 2024. arXiv:2402.17762. % [TO VERIFY]
|
||||
\bibitem{attnsink} ``When attention sink emerges in language models: an empirical
|
||||
view,'' 2024. arXiv:2410.10781. % [TO VERIFY authors]
|
||||
\bibitem{softcap} Gemma Team, ``Gemma 2,'' technical report, 2024. % [TO VERIFY] find arXiv ID + exact citation for logit soft-capping
|
||||
\bibitem{zloss} A. Chowdhery et al., ``PaLM: Scaling language modeling with
|
||||
pathways,'' 2022. % [TO VERIFY] z-loss source; confirm arXiv ID
|
||||
\bibitem{normformer} S. Shleifer et al., ``NormFormer: Improved transformer
|
||||
pretraining with extra normalization,'' 2021. arXiv:2110.09456. % [TO VERIFY authors]
|
||||
\bibitem{qknmla} ``QK-Normed MLA: QK normalization without full key caching,''
|
||||
2026. arXiv:2606.16310. % [TO VERIFY authors]
|
||||
\bibitem{tinystories} R. Eldan and Y. Li, ``TinyStories: How small can language
|
||||
models be and still speak coherent English?,'' 2023. arXiv:2305.07759. % [TO VERIFY]
|
||||
\bibitem{cramming} J. Geiping and T. Goldstein, ``Cramming: Training a language
|
||||
model on a single GPU in one day,'' in \textit{ICML}, 2023. arXiv:2212.14034. % [TO VERIFY]
|
||||
|
||||
\end{thebibliography}
|
||||
|
||||
\end{document}
|
||||
Reference in new issue
Block a user