167 lines
8.4 KiB
Markdown
167 lines
8.4 KiB
Markdown
---
|
||
marp: true
|
||
theme: default
|
||
paginate: true
|
||
size: 16:9
|
||
header: 'CS6513 · Research Proposal Pitch'
|
||
style: |
|
||
section { font-size: 26px; }
|
||
h1 { color: #1a3b5d; }
|
||
h2 { color: #1a3b5d; }
|
||
strong { color: #b0451f; }
|
||
section.title h1 { font-size: 46px; }
|
||
small { color: #666; }
|
||
---
|
||
|
||
<!-- _class: title -->
|
||
<!-- _paginate: false -->
|
||
<!-- _header: '' -->
|
||
|
||
# When Does a Small Model *Need* QK-Norm?
|
||
|
||
## A reproducible, isolated learning-rate characterization of Query–Key Normalization at <100M parameters
|
||
|
||
**Cormac Shannon** · ID 23366303
|
||
MSc Immersive Software Engineering · CS6513 Research Proposal Pitch
|
||
|
||
<small>Provisional working title — draft pitch</small>
|
||
|
||
<!-- notes: Hi, I'm Cormac. My study takes a popular, cheap architectural trick — QK-norm — and pins
|
||
down *exactly when* a small model actually needs it, with a clean, reproducible experiment. ~15s -->
|
||
|
||
---
|
||
|
||
## 1 · The Problem & Topic
|
||
|
||
- Modern LLMs stack many components — RoPE, RMSNorm, GQA, SwiGLU, **QK-norm** — mostly validated
|
||
at **large scale**. QK-norm normalises the query & key vectors so attention logits can't explode.
|
||
- **What's already known:** QK-norm improves training stability, *especially at high learning rates* —
|
||
shown by dense LR-sweeps but with QK-norm **bundled** with other fixes (Wortsman, ICLR 2024), and as
|
||
the **top single architectural win** in 70M ablations (IMU-1, 2026 — by cutting extreme activation outliers, *not* an LR sweep).
|
||
- **What's still fuzzy — my focus:** the *qualitative* rule is known; a **clean, dense measurement is not**.
|
||
Small-scale evidence is **patchy and inconsistently measured** (larger-scale proxies; looser signals —
|
||
extreme activation outliers, a single "≈1.5× LR" figure, a coarse 3-point blog sweep at 1B). **Un-charted: the dense
|
||
LR-sensitivity curve for *isolated* QK-norm at 10–100M, and how the LR at which it becomes necessary
|
||
moves with model depth.**
|
||
|
||
<!-- notes: Modern architectures pile on tricks proven on big models. QK-norm is one, and we already
|
||
know it helps stability at high learning rates — even in small models. What's NOT nailed down is the
|
||
precise small-scale picture: a clean dense learning-rate curve, and the exact point it becomes
|
||
necessary. That's my gap. ~50s -->
|
||
|
||
---
|
||
|
||
## 2 · Significance — the "So What?"
|
||
|
||
- **The field is going small** — on-device/edge, cost, energy, reproducibility all push toward
|
||
sub-100M LMs. Practitioners need concrete rules for *that* regime.
|
||
- **A usable rule of thumb:** if I can map the learning-rate (× depth) threshold where QK-norm flips
|
||
from *harmless-to-drop* to *necessary*, a small-model builder gets a direct answer — "below this LR
|
||
you can skip it; above it you can't." The qualitative flip is known; the **precise threshold — and how
|
||
it moves with depth — isn't charted at this scale.**
|
||
- **Evidence quality:** current small-scale results use *looser* metrics; a **dense LR-sensitivity
|
||
sweep** (à la Wortsman) puts the small-model claim on firmer footing.
|
||
- **Reproducibility:** a **laptop-runnable, open** artifact adds trustworthy, re-runnable evidence to a
|
||
literature dominated by expensive, hard-to-reproduce large-scale results.
|
||
|
||
<!-- notes: Why care? Small models are where deployment is heading. If I can pin the learning-rate and
|
||
depth threshold where QK-norm becomes necessary, that's a concrete rule builders can use — and nobody's
|
||
charted it. Plus a laptop-reproducible artifact strengthens shaky small-scale evidence. ~55s -->
|
||
|
||
---
|
||
|
||
## 3 · Conceptual Framework — Questions & Objectives
|
||
|
||
**Primary RQ.** At <100M parameters, does *isolated* QK-norm's stability benefit reproduce under a
|
||
**dense learning-rate sweep**, and **at what learning rate does it become necessary?**
|
||
|
||
**Sub-RQ.** How does that **necessity threshold shift with model depth?**
|
||
|
||
**Hypotheses**
|
||
- **H1** — QK-norm's effect is **LR-conditional**: negligible/slightly negative at low LR, decisive at high LR.
|
||
- **H2** — QK-norm **raises the max stable LR** and **lowers LR-sensitivity** at <100M.
|
||
- **H3** — the LR at which QK-norm becomes necessary **falls as depth increases** (deeper ⇒ needs it sooner).
|
||
|
||
**Objectives:** isolate a muP-free QK-norm toggle → dense LR sweep × a few depths → quantify
|
||
max-stable-LR, LR-sensitivity, activation outliers → **chart the LR × depth necessity boundary** → release the artifact.
|
||
|
||
<!-- notes: The core question isn't "does it help" — we know it can — it's *when*. Does the effect hold
|
||
under a proper dense sweep at small scale, at what learning rate does it become necessary, and how does
|
||
that move with depth. Three hypotheses built around that LR-conditionality. ~50s -->
|
||
|
||
---
|
||
|
||
## 4 · Methodological Approach
|
||
|
||
**Quantitative, experimental — a controlled ablation** (per Creswell, Ch. 1).
|
||
|
||
| Element | Choice |
|
||
|:--|:--|
|
||
| Independent variable | QK-norm **{on, off}** |
|
||
| Moderators | **Learning rate** (dense, log-spaced, safe → divergent) · **depth** (2–3 values) |
|
||
| Controls | identical width/data/tokens/optimizer (AdamW) · **μP explicitly OFF** · fixed seeds |
|
||
| Model | plain **nanoGPT** decoder, **~10–15M params** |
|
||
| Data | TinyStories / FineWeb-Edu slice (fixed token budget) |
|
||
| Measures | **max stable LR** · **LR-sensitivity** (Wortsman) · steps-to-threshold · perplexity · attention entropy · extreme activation outliers |
|
||
|
||
**Analysis:** loss-vs-LR curves per depth (≥2 seeds, mean ± spread); locate the divergence boundary and
|
||
the QK-norm necessity threshold. Base = **plain nanoGPT, μP off** so QK-norm is the *only* variable.
|
||
|
||
<!-- notes: A clean quantitative experiment: one thing changes — QK-norm on or off — across a dense
|
||
learning-rate sweep at a couple of depths, everything else fixed, and crucially with muP switched off so
|
||
I'm not confounding the learning-rate effect. ~55s -->
|
||
|
||
---
|
||
|
||
## 5 · Feasibility & Timeline
|
||
|
||
- **Runs on my hardware:** M4 Pro (48 GB), PyTorch **MPS**. ~10–15M model on ~100–150M tokens ≈
|
||
**~1–1.5 h/run**. **Core** matrix: QK{2} × LR{6} × seed{2} ≈ **24 runs** → 2–3 overnight batches.
|
||
- **Stretch** (the depth boundary): add 2 more depths near the threshold (~+18 runs) — or a rented
|
||
RTX 4090 (~\$0.40/hr) clears it in a few dollars.
|
||
- **Not** GPT-2 scale — an ablation needs *relative* effects, so tiny models are correct, not a compromise.
|
||
|
||
| Week | Milestone |
|
||
|:--|:--|
|
||
| **5** | Fork nanoGPT · isolated μP-off QK-norm toggle · reproduce baseline · pilot runs |
|
||
| **6** | Core dense LR sweep (≥2 seeds) · log curves |
|
||
| **7** | Analysis + (stretch) depth-threshold runs · figures |
|
||
| **8** | Write Literature Review + Introduction · finalise notebook |
|
||
|
||
<!-- notes: This fits my laptop: tiny models, about an hour a run, ~24 runs for the core over a couple
|
||
of nights, with the depth boundary as a stretch goal or a few dollars of cloud GPU. Four weeks, one
|
||
milestone each. ~50s -->
|
||
|
||
---
|
||
|
||
## 6 · Ethics & Data Access
|
||
|
||
- **No human subjects, no personal or sensitive data** → no ethical-approval barriers.
|
||
- **Public, openly-licensed datasets** (TinyStories, FineWeb-Edu) — licences to be confirmed and
|
||
respected; **no data-access constraints.**
|
||
- **Low environmental footprint** — a few GPU-hours total; energy-efficiency is itself a study motivation.
|
||
- **Reproducibility & integrity:** code, configs, seeds, and logs released openly; AI-tool use declared
|
||
in the research notebook per module policy.
|
||
|
||
<!-- notes: Ethically low-risk: no people, no private data, only open datasets. The compute footprint is
|
||
tiny — which is part of the point — and everything is released for reproducibility. ~35s -->
|
||
|
||
---
|
||
|
||
## 7 · Summary
|
||
|
||
- **Not "does QK-norm help" (known) — but *when a small model needs it*, charted cleanly.**
|
||
- A **dense, isolated, μP-free** LR sweep at <100M → the **max-stable-LR / LR-sensitivity** benefit *and*
|
||
the **LR × depth necessity boundary**, **reproducible on a laptop.**
|
||
- An honest **verification-plus-boundary-charting** study: firms up patchy small-scale evidence and
|
||
hands practitioners a usable rule.
|
||
|
||
### Thank you — questions welcome
|
||
|
||
<small>Key references: Henry et al. 2020 · Dehghani et al. 2023 (ViT-22B) · Wortsman et al. (ICLR 2024) ·
|
||
IMU-1 (2026) · OLMo 2 (2025) — *venues/details to verify against primary sources*</small>
|
||
|
||
<!-- notes: In short: not whether QK-norm helps — we know it can — but pinning down exactly when a small
|
||
model needs it, with a clean, reproducible, laptop-scale experiment that firms up shaky evidence and
|
||
gives builders a usable rule. Thank you. ~30s -->
|