---
marp: true
theme: default
paginate: true
size: 16:9
header: 'CS6513 · Research Proposal Pitch'
style: |
  section { font-size: 26px; }
  h1 { color: #1a3b5d; }
  h2 { color: #1a3b5d; }
  strong { color: #b0451f; }
  section.title h1 { font-size: 46px; }
  small { color: #666; }
---

<!-- _class: title -->
<!-- _paginate: false -->
<!-- _header: '' -->

# When Does a Small Model *Need* QK-Norm?

## A reproducible, isolated learning-rate characterization of Query–Key Normalization at <100M parameters

**Cormac Shannon** · ID 23366303
MSc Immersive Software Engineering · CS6513 Research Proposal Pitch

<small>Provisional working title — draft pitch</small>

<!-- notes: Hi, I'm Cormac. My study takes a popular, cheap architectural trick — QK-norm — and pins
down *exactly when* a small model actually needs it, with a clean, reproducible experiment. ~15s -->

---

## 1 · The Problem & Topic

- Modern LLMs stack many components — RoPE, RMSNorm, GQA, SwiGLU, **QK-norm** — mostly validated
  at **large scale**. QK-norm normalises the query & key vectors so attention logits can't explode.
- **What's already known:** QK-norm improves training stability, *especially at high learning rates* —
  shown by dense LR-sweeps but with QK-norm **bundled** with other fixes (Wortsman, ICLR 2024), and as
  the **top single architectural win** in 70M ablations (IMU-1, 2026 — by cutting extreme activation outliers, *not* an LR sweep).
- **What's still fuzzy — my focus:** the *qualitative* rule is known; a **clean, dense measurement is not**.
  Small-scale evidence is **patchy and inconsistently measured** (larger-scale proxies; looser signals —
  extreme activation outliers, a single "≈1.5× LR" figure, a coarse 3-point blog sweep at 1B). **Un-charted: the dense
  LR-sensitivity curve for *isolated* QK-norm at 10–100M, and how the LR at which it becomes necessary
  moves with model depth.**

<!-- notes: Modern architectures pile on tricks proven on big models. QK-norm is one, and we already
know it helps stability at high learning rates — even in small models. What's NOT nailed down is the
precise small-scale picture: a clean dense learning-rate curve, and the exact point it becomes
necessary. That's my gap. ~50s -->

---

## 2 · Significance — the "So What?"

- **The field is going small** — on-device/edge, cost, energy, reproducibility all push toward
  sub-100M LMs. Practitioners need concrete rules for *that* regime.
- **A usable rule of thumb:** if I can map the learning-rate (× depth) threshold where QK-norm flips
  from *harmless-to-drop* to *necessary*, a small-model builder gets a direct answer — "below this LR
  you can skip it; above it you can't." The qualitative flip is known; the **precise threshold — and how
  it moves with depth — isn't charted at this scale.**
- **Evidence quality:** current small-scale results use *looser* metrics; a **dense LR-sensitivity
  sweep** (à la Wortsman) puts the small-model claim on firmer footing.
- **Reproducibility:** a **laptop-runnable, open** artifact adds trustworthy, re-runnable evidence to a
  literature dominated by expensive, hard-to-reproduce large-scale results.

<!-- notes: Why care? Small models are where deployment is heading. If I can pin the learning-rate and
depth threshold where QK-norm becomes necessary, that's a concrete rule builders can use — and nobody's
charted it. Plus a laptop-reproducible artifact strengthens shaky small-scale evidence. ~55s -->

---

## 3 · Conceptual Framework — Questions & Objectives

**Primary RQ.** At <100M parameters, does *isolated* QK-norm's stability benefit reproduce under a
**dense learning-rate sweep**, and **at what learning rate does it become necessary?**

**Sub-RQ.** How does that **necessity threshold shift with model depth?**

**Hypotheses**
- **H1** — QK-norm's effect is **LR-conditional**: negligible/slightly negative at low LR, decisive at high LR.
- **H2** — QK-norm **raises the max stable LR** and **lowers LR-sensitivity** at <100M.
- **H3** — the LR at which QK-norm becomes necessary **falls as depth increases** (deeper ⇒ needs it sooner).

**Objectives:** isolate a muP-free QK-norm toggle → dense LR sweep × a few depths → quantify
max-stable-LR, LR-sensitivity, activation outliers → **chart the LR × depth necessity boundary** → release the artifact.

<!-- notes: The core question isn't "does it help" — we know it can — it's *when*. Does the effect hold
under a proper dense sweep at small scale, at what learning rate does it become necessary, and how does
that move with depth. Three hypotheses built around that LR-conditionality. ~50s -->

---

## 4 · Methodological Approach

**Quantitative, experimental — a controlled ablation** (per Creswell, Ch. 1).

| Element | Choice |
|:--|:--|
| Independent variable | QK-norm **{on, off}** |
| Moderators | **Learning rate** (dense, log-spaced, safe → divergent) · **depth** (2–3 values) |
| Controls | identical width/data/tokens/optimizer (AdamW) · **μP explicitly OFF** · fixed seeds |
| Model | plain **nanoGPT** decoder, **~10–15M params** |
| Data | TinyStories / FineWeb-Edu slice (fixed token budget) |
| Measures | **max stable LR** · **LR-sensitivity** (Wortsman) · steps-to-threshold · perplexity · attention entropy · extreme activation outliers |

**Analysis:** loss-vs-LR curves per depth (≥2 seeds, mean ± spread); locate the divergence boundary and
the QK-norm necessity threshold. Base = **plain nanoGPT, μP off** so QK-norm is the *only* variable.

<!-- notes: A clean quantitative experiment: one thing changes — QK-norm on or off — across a dense
learning-rate sweep at a couple of depths, everything else fixed, and crucially with muP switched off so
I'm not confounding the learning-rate effect. ~55s -->

---

## 5 · Feasibility & Timeline

- **Runs on my hardware:** M4 Pro (48 GB), PyTorch **MPS**. ~10–15M model on ~100–150M tokens ≈
  **~1–1.5 h/run**. **Core** matrix: QK{2} × LR{6} × seed{2} ≈ **24 runs** → 2–3 overnight batches.
- **Stretch** (the depth boundary): add 2 more depths near the threshold (~+18 runs) — or a rented
  RTX 4090 (~\$0.40/hr) clears it in a few dollars.
- **Not** GPT-2 scale — an ablation needs *relative* effects, so tiny models are correct, not a compromise.

| Week | Milestone |
|:--|:--|
| **5** | Fork nanoGPT · isolated μP-off QK-norm toggle · reproduce baseline · pilot runs |
| **6** | Core dense LR sweep (≥2 seeds) · log curves |
| **7** | Analysis + (stretch) depth-threshold runs · figures |
| **8** | Write Literature Review + Introduction · finalise notebook |

<!-- notes: This fits my laptop: tiny models, about an hour a run, ~24 runs for the core over a couple
of nights, with the depth boundary as a stretch goal or a few dollars of cloud GPU. Four weeks, one
milestone each. ~50s -->

---

## 6 · Ethics & Data Access

- **No human subjects, no personal or sensitive data** → no ethical-approval barriers.
- **Public, openly-licensed datasets** (TinyStories, FineWeb-Edu) — licences to be confirmed and
  respected; **no data-access constraints.**
- **Low environmental footprint** — a few GPU-hours total; energy-efficiency is itself a study motivation.
- **Reproducibility & integrity:** code, configs, seeds, and logs released openly; AI-tool use declared
  in the research notebook per module policy.

<!-- notes: Ethically low-risk: no people, no private data, only open datasets. The compute footprint is
tiny — which is part of the point — and everything is released for reproducibility. ~35s -->

---

## 7 · Summary

- **Not "does QK-norm help" (known) — but *when a small model needs it*, charted cleanly.**
- A **dense, isolated, μP-free** LR sweep at <100M → the **max-stable-LR / LR-sensitivity** benefit *and*
  the **LR × depth necessity boundary**, **reproducible on a laptop.**
- An honest **verification-plus-boundary-charting** study: firms up patchy small-scale evidence and
  hands practitioners a usable rule.

### Thank you — questions welcome

<small>Key references: Henry et al. 2020 · Dehghani et al. 2023 (ViT-22B) · Wortsman et al. (ICLR 2024) ·
IMU-1 (2026) · OLMo 2 (2025) — *venues/details to verify against primary sources*</small>

<!-- notes: In short: not whether QK-norm helps — we know it can — but pinning down exactly when a small
model needs it, with a clean, reproducible, laptop-scale experiment that firms up shaky evidence and
gives builders a usable rule. Thank you. ~30s -->