6.3 KiB
5-Minute Pitch — Narration Script (DRAFT)
How to use this: this is a scaffold in your voice — read it aloud, cut what feels unnatural, and make the wording your own before recording. The video is you presenting, so re-voicing it is expected. Log this AI-assisted drafting in the notebook's GenAI Transparency Log (Section 3).
Framing note: this is an honest verification + boundary-charting study, not a novelty claim. Do not say "first," "nobody has studied QK-norm at small scale," or "resolves a contradiction" — a panel that knows Wortsman (ICLR 2024) or IMU-1 (2026) would push back. The defensible line is pinning down when a small model needs QK-norm, with a clean reproducible sweep.
Target ≈ 700 words ≈ 5 min at a calm pace. Timings are cumulative.
[Title — 0:00–0:15] Hi, I'm Cormac Shannon. We already know that a popular, cheap architectural trick called QK-norm helps stabilize language-model training. My study asks a sharper, unanswered question: when does a small model actually need it — and I'll answer that with a clean, reproducible experiment.
[Slide 1 · Problem & Topic — 0:15–1:05] Modern language models are a stack of architectural components — rotary embeddings, RMSNorm, grouped-query attention, QK-norm, and more. QK-norm simply normalizes the query and key vectors before the attention step, so the attention scores can't blow up. Now, it's established that this helps stability, especially at high learning rates — and that even holds on small models; a 2023 study swept learning rates at small scale, and in a 2026 paper QK-norm was the single biggest architectural win in seventy-million-parameter ablations. So I'm not claiming to discover that — the qualitative rule is already known. What's genuinely fuzzy is the small-scale detail: the evidence is patchy and measured inconsistently — some at larger scale, some using looser signals like activation spikiness or a single "roughly one-and-a-half-times learning rate" number. What hasn't been done is a clean, dense learning-rate curve for QK-norm in isolation at ten to a hundred million parameters, and mapping how the point where it becomes necessary shifts as the model gets deeper.
[Slide 2 · Significance — 1:05–1:55] Why does this matter? First, the field is moving toward small models — for on-device use, cost, energy, and reproducibility — yet the design rules for that regime are under-studied. Second, and this is the real payoff: if I can map the learning-rate and depth threshold where QK-norm becomes necessary, a small-model builder gets a concrete rule — below this learning rate you can drop it, above it you can't. The qualitative flip is known, but that precise threshold — and how it shifts with depth — hasn't been charted at this scale. Third, on evidence quality: a proper dense learning-rate sweep puts the shaky small-scale claims on firmer footing. And finally, because the whole thing runs on a laptop, I can release it openly — adding reproducible, re-runnable evidence to a literature dominated by expensive results almost nobody can reproduce.
[Slide 3 · Conceptual Framework — 1:55–2:45] So here's my framework. My primary question: at under a hundred million parameters, does isolated QK-norm's stability benefit reproduce under a dense learning-rate sweep — and at what learning rate does it actually become necessary? My sub-question: how does that necessity threshold shift as the model gets deeper? That gives three testable hypotheses. One: the benefit is learning-rate-conditional — negligible, or even slightly negative, at low learning rates, but decisive at high ones. Two: QK-norm raises the maximum stable learning rate and lowers learning-rate sensitivity. Three: the threshold where it becomes necessary falls as depth increases — deeper models need it sooner.
[Slide 4 · Methodology — 2:45–3:35]
Methodologically this is a quantitative, controlled ablation, in Creswell's terms. One independent
variable: QK-norm on or off. Two moderators: the learning rate, swept densely from safe to divergent,
and model depth at a couple of values. Everything else is held fixed — width, data, tokens, optimizer,
seeds — and, importantly, I switch off maximal-update parametrization, because that technique
transfers learning rates by design and would confound my result. The model is a plain nanoGPT-style
transformer, ten to fifteen million parameters, on an open dataset like TinyStories. I measure the
maximum stable learning rate, learning-rate sensitivity, steps-to-target-loss, perplexity, and — as
mechanistic checks — attention entropy and activation spikiness, repeated over at least two seeds.
[Slide 5 · Feasibility & Timeline — 3:35–4:20] On feasibility: this fits my hardware — an M4 Pro with forty-eight gigabytes, using PyTorch's MPS backend. A model this size takes roughly an hour to an hour and a half per run. My core matrix — QK-norm on or off, six learning rates, two seeds — is about twenty-four runs, a couple of overnight batches. The depth-threshold part is a stretch goal, a few more runs, or a rented GPU for a few dollars. I'm deliberately not training at GPT-2 scale, because an ablation only needs relative effects, so tiny models are the correct choice. The timeline is one milestone a week: implementation and baseline in week five, the core sweep in week six, analysis and the depth runs in week seven, and writing up in week eight.
[Slide 6 · Ethics & Data — 4:20–4:45] Ethically this is low-risk. No human subjects and no personal or sensitive data, so no approval barriers. I'm using openly-licensed public datasets with no access constraints. The compute footprint is only a few GPU-hours — and energy efficiency is actually part of my motivation. And everything — code, configs, seeds, logs — will be released for reproducibility.
[Slide 7 · Summary — 4:45–5:00] So, in one line: not whether QK-norm helps — we know it can — but pinning down exactly when a small model needs it, with a clean, reproducible, laptop-scale experiment that firms up shaky evidence and hands practitioners a usable rule. Thank you, and I'm happy to take questions.
Word count ≈ 720. If you run long, the easiest trims are the third point of Slide 2 (evidence quality) and the mechanistic-checks detail on Slide 4.