84 lines
6.3 KiB
Markdown
84 lines
6.3 KiB
Markdown
# 5-Minute Pitch — Narration Script (DRAFT)
|
||
|
||
**How to use this:** this is a *scaffold in your voice* — read it aloud, cut what feels unnatural,
|
||
and make the wording your own before recording. The video is *you* presenting, so re-voicing it is
|
||
expected. Log this AI-assisted drafting in the notebook's GenAI Transparency Log (Section 3).
|
||
|
||
**Framing note:** this is an honest **verification + boundary-charting** study, *not* a novelty claim.
|
||
Do not say "first," "nobody has studied QK-norm at small scale," or "resolves a contradiction" — a
|
||
panel that knows Wortsman (ICLR 2024) or IMU-1 (2026) would push back. The defensible line is *pinning down
|
||
**when** a small model needs QK-norm, with a clean reproducible sweep.*
|
||
|
||
Target ≈ 700 words ≈ 5 min at a calm pace. Timings are cumulative.
|
||
|
||
---
|
||
|
||
**[Title — 0:00–0:15]**
|
||
Hi, I'm Cormac Shannon. We already know that a popular, cheap architectural trick called QK-norm helps
|
||
stabilize language-model training. My study asks a sharper, unanswered question: *when* does a small
|
||
model actually need it — and I'll answer that with a clean, reproducible experiment.
|
||
|
||
**[Slide 1 · Problem & Topic — 0:15–1:05]** Modern language models are a stack of architectural components — rotary embeddings, RMSNorm, grouped-query attention, QK-norm, and more. QK-norm simply normalizes the query and key vectors before the attention step, so the attention scores can't blow up. Now, it's *established* that this helps
|
||
stability, especially at high learning rates — and that even holds on small models; a 2023 study swept
|
||
learning rates at small scale, and in a 2026 paper QK-norm was the single biggest architectural win in
|
||
seventy-million-parameter ablations. So I'm not claiming to discover that — the qualitative rule is
|
||
already known. What's genuinely fuzzy is the *small-scale detail*: the evidence is patchy and measured
|
||
inconsistently — some at larger scale, some using looser signals like activation spikiness or a single
|
||
"roughly one-and-a-half-times learning rate" number. What hasn't been done is a clean, dense
|
||
learning-rate curve for QK-norm in isolation at ten to a hundred million parameters, and mapping how the
|
||
point where it becomes necessary shifts as the model gets deeper.
|
||
|
||
**[Slide 2 · Significance — 1:05–1:55]**
|
||
Why does this matter? First, the field is moving toward small models — for on-device use, cost, energy,
|
||
and reproducibility — yet the design rules for that regime are under-studied. Second, and this is the
|
||
real payoff: if I can map the learning-rate and depth threshold where QK-norm becomes necessary, a
|
||
small-model builder gets a concrete rule — below this learning rate you can drop it, above it you can't.
|
||
The qualitative flip is known, but that precise threshold — and how it shifts with depth — hasn't been
|
||
charted at this scale. Third, on evidence quality: a proper dense learning-rate
|
||
sweep puts the shaky small-scale claims on firmer footing. And finally, because the whole thing runs on
|
||
a laptop, I can release it openly — adding reproducible, re-runnable evidence to a literature dominated
|
||
by expensive results almost nobody can reproduce.
|
||
|
||
**[Slide 3 · Conceptual Framework — 1:55–2:45]**
|
||
So here's my framework. My primary question: at under a hundred million parameters, does isolated
|
||
QK-norm's stability benefit reproduce under a dense learning-rate sweep — and at what learning rate does
|
||
it actually become necessary? My sub-question: how does that necessity threshold shift as the model gets
|
||
deeper? That gives three testable hypotheses. One: the benefit is *learning-rate-conditional* —
|
||
negligible, or even slightly negative, at low learning rates, but decisive at high ones. Two: QK-norm
|
||
raises the maximum stable learning rate and lowers learning-rate sensitivity. Three: the threshold where
|
||
it becomes necessary falls as depth increases — deeper models need it sooner.
|
||
|
||
**[Slide 4 · Methodology — 2:45–3:35]**
|
||
Methodologically this is a quantitative, controlled ablation, in Creswell's terms. One independent
|
||
variable: QK-norm on or off. Two moderators: the learning rate, swept densely from safe to divergent,
|
||
and model depth at a couple of values. Everything else is held fixed — width, data, tokens, optimizer,
|
||
seeds — ~~and, importantly, I switch *off* maximal-update parametrization, because that technique
|
||
transfers learning rates by design and would confound my result~~. The model is a plain nanoGPT-style
|
||
transformer, ten to fifteen million parameters, on an open dataset like TinyStories. I measure the
|
||
maximum stable learning rate, learning-rate sensitivity, steps-to-target-loss, perplexity, and — as
|
||
mechanistic checks — attention entropy and activation spikiness, repeated over at least two seeds.
|
||
|
||
**[Slide 5 · Feasibility & Timeline — 3:35–4:20]**
|
||
On feasibility: this fits my hardware — an M4 Pro with forty-eight gigabytes, using PyTorch's MPS
|
||
backend. A model this size takes roughly an hour to an hour and a half per run. My core matrix — QK-norm
|
||
on or off, six learning rates, two seeds — is about twenty-four runs, a couple of overnight batches. The
|
||
depth-threshold part is a stretch goal, a few more runs, or a rented GPU for a few dollars. I'm
|
||
deliberately *not* training at GPT-2 scale, because an ablation only needs relative effects, so tiny
|
||
models are the correct choice. The timeline is one milestone a week: implementation and baseline in week
|
||
five, the core sweep in week six, analysis and the depth runs in week seven, and writing up in week eight.
|
||
|
||
**[Slide 6 · Ethics & Data — 4:20–4:45]**
|
||
Ethically this is low-risk. No human subjects and no personal or sensitive data, so no approval
|
||
barriers. I'm using openly-licensed public datasets with no access constraints. The compute footprint is
|
||
only a few GPU-hours — and energy efficiency is actually part of my motivation. And everything — code,
|
||
configs, seeds, logs — will be released for reproducibility.
|
||
|
||
**[Slide 7 · Summary — 4:45–5:00]**
|
||
So, in one line: not *whether* QK-norm helps — we know it can — but pinning down exactly *when* a small
|
||
model needs it, with a clean, reproducible, laptop-scale experiment that firms up shaky evidence and
|
||
hands practitioners a usable rule. Thank you, and I'm happy to take questions.
|
||
|
||
---
|
||
|
||
*Word count ≈ 720. If you run long, the easiest trims are the third point of Slide 2 (evidence quality)
|
||
and the mechanistic-checks detail on Slide 4.* |