Organise MSc thesis project into one folder

01-proposal/pitch-script.md

@@ -0,0 +1,84 @@
+# 5-Minute Pitch — Narration Script (DRAFT)
+
+**How to use this:** this is a *scaffold in your voice* — read it aloud, cut what feels unnatural,
+and make the wording your own before recording. The video is *you* presenting, so re-voicing it is
+expected. Log this AI-assisted drafting in the notebook's GenAI Transparency Log (Section 3).
+
+**Framing note:** this is an honest **verification + boundary-charting** study, *not* a novelty claim.
+Do not say "first," "nobody has studied QK-norm at small scale," or "resolves a contradiction" — a
+panel that knows Wortsman (ICLR 2024) or IMU-1 (2026) would push back. The defensible line is *pinning down
+**when** a small model needs QK-norm, with a clean reproducible sweep.*
+
+Target ≈ 700 words ≈ 5 min at a calm pace. Timings are cumulative.
+
+---
+
+**[Title — 0:00–0:15]**
+Hi, I'm Cormac Shannon. We already know that a popular, cheap architectural trick called QK-norm helps
+stabilize language-model training. My study asks a sharper, unanswered question: *when* does a small
+model actually need it — and I'll answer that with a clean, reproducible experiment.
+
+**[Slide 1 · Problem & Topic — 0:15–1:05]** Modern language models are a stack of architectural components — rotary embeddings, RMSNorm, grouped-query attention, QK-norm, and more. QK-norm simply normalizes the query and key vectors before the attention step, so the attention scores can't blow up. Now, it's *established* that this helps
+stability, especially at high learning rates — and that even holds on small models; a 2023 study swept
+learning rates at small scale, and in a 2026 paper QK-norm was the single biggest architectural win in
+seventy-million-parameter ablations. So I'm not claiming to discover that — the qualitative rule is
+already known. What's genuinely fuzzy is the *small-scale detail*: the evidence is patchy and measured
+inconsistently — some at larger scale, some using looser signals like activation spikiness or a single 
+"roughly one-and-a-half-times learning rate" number. What hasn't been done is a clean, dense
+learning-rate curve for QK-norm in isolation at ten to a hundred million parameters, and mapping how the
+point where it becomes necessary shifts as the model gets deeper.
+
+**[Slide 2 · Significance — 1:05–1:55]**
+Why does this matter? First, the field is moving toward small models — for on-device use, cost, energy,
+and reproducibility — yet the design rules for that regime are under-studied. Second, and this is the
+real payoff: if I can map the learning-rate and depth threshold where QK-norm becomes necessary, a
+small-model builder gets a concrete rule — below this learning rate you can drop it, above it you can't.
+The qualitative flip is known, but that precise threshold — and how it shifts with depth — hasn't been
+charted at this scale. Third, on evidence quality: a proper dense learning-rate
+sweep puts the shaky small-scale claims on firmer footing. And finally, because the whole thing runs on
+a laptop, I can release it openly — adding reproducible, re-runnable evidence to a literature dominated
+by expensive results almost nobody can reproduce.
+
+**[Slide 3 · Conceptual Framework — 1:55–2:45]**
+So here's my framework. My primary question: at under a hundred million parameters, does isolated
+QK-norm's stability benefit reproduce under a dense learning-rate sweep — and at what learning rate does
+it actually become necessary? My sub-question: how does that necessity threshold shift as the model gets
+deeper? That gives three testable hypotheses. One: the benefit is *learning-rate-conditional* —
+negligible, or even slightly negative, at low learning rates, but decisive at high ones. Two: QK-norm
+raises the maximum stable learning rate and lowers learning-rate sensitivity. Three: the threshold where
+it becomes necessary falls as depth increases — deeper models need it sooner.
+
+**[Slide 4 · Methodology — 2:45–3:35]**
+Methodologically this is a quantitative, controlled ablation, in Creswell's terms. One independent
+variable: QK-norm on or off. Two moderators: the learning rate, swept densely from safe to divergent,
+and model depth at a couple of values. Everything else is held fixed — width, data, tokens, optimizer,
+seeds — ~~and, importantly, I switch *off* maximal-update parametrization, because that technique
+transfers learning rates by design and would confound my result~~. The model is a plain nanoGPT-style
+transformer, ten to fifteen million parameters, on an open dataset like TinyStories. I measure the
+maximum stable learning rate, learning-rate sensitivity, steps-to-target-loss, perplexity, and — as
+mechanistic checks — attention entropy and activation spikiness, repeated over at least two seeds.
+
+**[Slide 5 · Feasibility & Timeline — 3:35–4:20]**
+On feasibility: this fits my hardware — an M4 Pro with forty-eight gigabytes, using PyTorch's MPS
+backend. A model this size takes roughly an hour to an hour and a half per run. My core matrix — QK-norm
+on or off, six learning rates, two seeds — is about twenty-four runs, a couple of overnight batches. The
+depth-threshold part is a stretch goal, a few more runs, or a rented GPU for a few dollars. I'm
+deliberately *not* training at GPT-2 scale, because an ablation only needs relative effects, so tiny
+models are the correct choice. The timeline is one milestone a week: implementation and baseline in week
+five, the core sweep in week six, analysis and the depth runs in week seven, and writing up in week eight.
+
+**[Slide 6 · Ethics & Data — 4:20–4:45]**
+Ethically this is low-risk. No human subjects and no personal or sensitive data, so no approval
+barriers. I'm using openly-licensed public datasets with no access constraints. The compute footprint is
+only a few GPU-hours — and energy efficiency is actually part of my motivation. And everything — code,
+configs, seeds, logs — will be released for reproducibility.
+
+**[Slide 7 · Summary — 4:45–5:00]**
+So, in one line: not *whether* QK-norm helps — we know it can — but pinning down exactly *when* a small
+model needs it, with a clean, reproducible, laptop-scale experiment that firms up shaky evidence and
+hands practitioners a usable rule. Thank you, and I'm happy to take questions.
+
+---
+
+*Word count ≈ 720. If you run long, the easiest trims are the third point of Slide 2 (evidence quality)
+and the mechanistic-checks detail on Slide 4.*