Organise MSc thesis project into one folder

01-proposal/pitch-script.md

@@ -0,0 +1,84 @@
+# 5-Minute Pitch — Narration Script (DRAFT)
+
+**How to use this:** this is a *scaffold in your voice* — read it aloud, cut what feels unnatural,
+and make the wording your own before recording. The video is *you* presenting, so re-voicing it is
+expected. Log this AI-assisted drafting in the notebook's GenAI Transparency Log (Section 3).
+
+**Framing note:** this is an honest **verification + boundary-charting** study, *not* a novelty claim.
+Do not say "first," "nobody has studied QK-norm at small scale," or "resolves a contradiction" — a
+panel that knows Wortsman (ICLR 2024) or IMU-1 (2026) would push back. The defensible line is *pinning down
+**when** a small model needs QK-norm, with a clean reproducible sweep.*
+
+Target ≈ 700 words ≈ 5 min at a calm pace. Timings are cumulative.
+
+---
+
+**[Title — 0:00–0:15]**
+Hi, I'm Cormac Shannon. We already know that a popular, cheap architectural trick called QK-norm helps
+stabilize language-model training. My study asks a sharper, unanswered question: *when* does a small
+model actually need it — and I'll answer that with a clean, reproducible experiment.
+
+**[Slide 1 · Problem & Topic — 0:15–1:05]** Modern language models are a stack of architectural components — rotary embeddings, RMSNorm, grouped-query attention, QK-norm, and more. QK-norm simply normalizes the query and key vectors before the attention step, so the attention scores can't blow up. Now, it's *established* that this helps
+stability, especially at high learning rates — and that even holds on small models; a 2023 study swept
+learning rates at small scale, and in a 2026 paper QK-norm was the single biggest architectural win in
+seventy-million-parameter ablations. So I'm not claiming to discover that — the qualitative rule is
+already known. What's genuinely fuzzy is the *small-scale detail*: the evidence is patchy and measured
+inconsistently — some at larger scale, some using looser signals like activation spikiness or a single 
+"roughly one-and-a-half-times learning rate" number. What hasn't been done is a clean, dense
+learning-rate curve for QK-norm in isolation at ten to a hundred million parameters, and mapping how the
+point where it becomes necessary shifts as the model gets deeper.
+
+**[Slide 2 · Significance — 1:05–1:55]**
+Why does this matter? First, the field is moving toward small models — for on-device use, cost, energy,
+and reproducibility — yet the design rules for that regime are under-studied. Second, and this is the
+real payoff: if I can map the learning-rate and depth threshold where QK-norm becomes necessary, a
+small-model builder gets a concrete rule — below this learning rate you can drop it, above it you can't.
+The qualitative flip is known, but that precise threshold — and how it shifts with depth — hasn't been
+charted at this scale. Third, on evidence quality: a proper dense learning-rate
+sweep puts the shaky small-scale claims on firmer footing. And finally, because the whole thing runs on
+a laptop, I can release it openly — adding reproducible, re-runnable evidence to a literature dominated
+by expensive results almost nobody can reproduce.
+
+**[Slide 3 · Conceptual Framework — 1:55–2:45]**
+So here's my framework. My primary question: at under a hundred million parameters, does isolated
+QK-norm's stability benefit reproduce under a dense learning-rate sweep — and at what learning rate does
+it actually become necessary? My sub-question: how does that necessity threshold shift as the model gets
+deeper? That gives three testable hypotheses. One: the benefit is *learning-rate-conditional* —
+negligible, or even slightly negative, at low learning rates, but decisive at high ones. Two: QK-norm
+raises the maximum stable learning rate and lowers learning-rate sensitivity. Three: the threshold where
+it becomes necessary falls as depth increases — deeper models need it sooner.
+
+**[Slide 4 · Methodology — 2:45–3:35]**
+Methodologically this is a quantitative, controlled ablation, in Creswell's terms. One independent
+variable: QK-norm on or off. Two moderators: the learning rate, swept densely from safe to divergent,
+and model depth at a couple of values. Everything else is held fixed — width, data, tokens, optimizer,
+seeds — ~~and, importantly, I switch *off* maximal-update parametrization, because that technique
+transfers learning rates by design and would confound my result~~. The model is a plain nanoGPT-style
+transformer, ten to fifteen million parameters, on an open dataset like TinyStories. I measure the
+maximum stable learning rate, learning-rate sensitivity, steps-to-target-loss, perplexity, and — as
+mechanistic checks — attention entropy and activation spikiness, repeated over at least two seeds.
+
+**[Slide 5 · Feasibility & Timeline — 3:35–4:20]**
+On feasibility: this fits my hardware — an M4 Pro with forty-eight gigabytes, using PyTorch's MPS
+backend. A model this size takes roughly an hour to an hour and a half per run. My core matrix — QK-norm
+on or off, six learning rates, two seeds — is about twenty-four runs, a couple of overnight batches. The
+depth-threshold part is a stretch goal, a few more runs, or a rented GPU for a few dollars. I'm
+deliberately *not* training at GPT-2 scale, because an ablation only needs relative effects, so tiny
+models are the correct choice. The timeline is one milestone a week: implementation and baseline in week
+five, the core sweep in week six, analysis and the depth runs in week seven, and writing up in week eight.
+
+**[Slide 6 · Ethics & Data — 4:20–4:45]**
+Ethically this is low-risk. No human subjects and no personal or sensitive data, so no approval
+barriers. I'm using openly-licensed public datasets with no access constraints. The compute footprint is
+only a few GPU-hours — and energy efficiency is actually part of my motivation. And everything — code,
+configs, seeds, logs — will be released for reproducibility.
+
+**[Slide 7 · Summary — 4:45–5:00]**
+So, in one line: not *whether* QK-norm helps — we know it can — but pinning down exactly *when* a small
+model needs it, with a clean, reproducible, laptop-scale experiment that firms up shaky evidence and
+hands practitioners a usable rule. Thank you, and I'm happy to take questions.
+
+---
+
+*Word count ≈ 720. If you run long, the easiest trims are the third point of Slide 2 (evidence quality)
+and the mechanistic-checks detail on Slide 4.*

01-proposal/pitch-slides.md

@@ -0,0 +1,166 @@
+---
+marp: true
+theme: default
+paginate: true
+size: 16:9
+header: 'CS6513 · Research Proposal Pitch'
+style: |
+  section { font-size: 26px; }
+  h1 { color: #1a3b5d; }
+  h2 { color: #1a3b5d; }
+  strong { color: #b0451f; }
+  section.title h1 { font-size: 46px; }
+  small { color: #666; }
+---
+
+<!-- _class: title -->
+<!-- _paginate: false -->
+<!-- _header: '' -->
+
+# When Does a Small Model *Need* QK-Norm?
+
+## A reproducible, isolated learning-rate characterization of Query–Key Normalization at <100M parameters
+
+**Cormac Shannon** · ID 23366303
+MSc Immersive Software Engineering · CS6513 Research Proposal Pitch
+
+<small>Provisional working title — draft pitch</small>
+
+<!-- notes: Hi, I'm Cormac. My study takes a popular, cheap architectural trick — QK-norm — and pins
+down *exactly when* a small model actually needs it, with a clean, reproducible experiment. ~15s -->
+
+---
+
+## 1 · The Problem & Topic
+
+- Modern LLMs stack many components — RoPE, RMSNorm, GQA, SwiGLU, **QK-norm** — mostly validated
+  at **large scale**. QK-norm normalises the query & key vectors so attention logits can't explode.
+- **What's already known:** QK-norm improves training stability, *especially at high learning rates* —
+  shown by dense LR-sweeps but with QK-norm **bundled** with other fixes (Wortsman, ICLR 2024), and as
+  the **top single architectural win** in 70M ablations (IMU-1, 2026 — by cutting extreme activation outliers, *not* an LR sweep).
+- **What's still fuzzy — my focus:** the *qualitative* rule is known; a **clean, dense measurement is not**.
+  Small-scale evidence is **patchy and inconsistently measured** (larger-scale proxies; looser signals —
+  extreme activation outliers, a single "≈1.5× LR" figure, a coarse 3-point blog sweep at 1B). **Un-charted: the dense
+  LR-sensitivity curve for *isolated* QK-norm at 10–100M, and how the LR at which it becomes necessary
+  moves with model depth.**
+
+<!-- notes: Modern architectures pile on tricks proven on big models. QK-norm is one, and we already
+know it helps stability at high learning rates — even in small models. What's NOT nailed down is the
+precise small-scale picture: a clean dense learning-rate curve, and the exact point it becomes
+necessary. That's my gap. ~50s -->
+
+---
+
+## 2 · Significance — the "So What?"
+
+- **The field is going small** — on-device/edge, cost, energy, reproducibility all push toward
+  sub-100M LMs. Practitioners need concrete rules for *that* regime.
+- **A usable rule of thumb:** if I can map the learning-rate (× depth) threshold where QK-norm flips
+  from *harmless-to-drop* to *necessary*, a small-model builder gets a direct answer — "below this LR
+  you can skip it; above it you can't." The qualitative flip is known; the **precise threshold — and how
+  it moves with depth — isn't charted at this scale.**
+- **Evidence quality:** current small-scale results use *looser* metrics; a **dense LR-sensitivity
+  sweep** (à la Wortsman) puts the small-model claim on firmer footing.
+- **Reproducibility:** a **laptop-runnable, open** artifact adds trustworthy, re-runnable evidence to a
+  literature dominated by expensive, hard-to-reproduce large-scale results.
+
+<!-- notes: Why care? Small models are where deployment is heading. If I can pin the learning-rate and
+depth threshold where QK-norm becomes necessary, that's a concrete rule builders can use — and nobody's
+charted it. Plus a laptop-reproducible artifact strengthens shaky small-scale evidence. ~55s -->
+
+---
+
+## 3 · Conceptual Framework — Questions & Objectives
+
+**Primary RQ.** At <100M parameters, does *isolated* QK-norm's stability benefit reproduce under a
+**dense learning-rate sweep**, and **at what learning rate does it become necessary?**
+
+**Sub-RQ.** How does that **necessity threshold shift with model depth?**
+
+**Hypotheses**
+- **H1** — QK-norm's effect is **LR-conditional**: negligible/slightly negative at low LR, decisive at high LR.
+- **H2** — QK-norm **raises the max stable LR** and **lowers LR-sensitivity** at <100M.
+- **H3** — the LR at which QK-norm becomes necessary **falls as depth increases** (deeper ⇒ needs it sooner).
+
+**Objectives:** isolate a muP-free QK-norm toggle → dense LR sweep × a few depths → quantify
+max-stable-LR, LR-sensitivity, activation outliers → **chart the LR × depth necessity boundary** → release the artifact.
+
+<!-- notes: The core question isn't "does it help" — we know it can — it's *when*. Does the effect hold
+under a proper dense sweep at small scale, at what learning rate does it become necessary, and how does
+that move with depth. Three hypotheses built around that LR-conditionality. ~50s -->
+
+---
+
+## 4 · Methodological Approach
+
+**Quantitative, experimental — a controlled ablation** (per Creswell, Ch. 1).
+
+| Element | Choice |
+|:--|:--|
+| Independent variable | QK-norm **{on, off}** |
+| Moderators | **Learning rate** (dense, log-spaced, safe → divergent) · **depth** (2–3 values) |
+| Controls | identical width/data/tokens/optimizer (AdamW) · **μP explicitly OFF** · fixed seeds |
+| Model | plain **nanoGPT** decoder, **~10–15M params** |
+| Data | TinyStories / FineWeb-Edu slice (fixed token budget) |
+| Measures | **max stable LR** · **LR-sensitivity** (Wortsman) · steps-to-threshold · perplexity · attention entropy · extreme activation outliers |
+
+**Analysis:** loss-vs-LR curves per depth (≥2 seeds, mean ± spread); locate the divergence boundary and
+the QK-norm necessity threshold. Base = **plain nanoGPT, μP off** so QK-norm is the *only* variable.
+
+<!-- notes: A clean quantitative experiment: one thing changes — QK-norm on or off — across a dense
+learning-rate sweep at a couple of depths, everything else fixed, and crucially with muP switched off so
+I'm not confounding the learning-rate effect. ~55s -->
+
+---
+
+## 5 · Feasibility & Timeline
+
+- **Runs on my hardware:** M4 Pro (48 GB), PyTorch **MPS**. ~10–15M model on ~100–150M tokens ≈
+  **~1–1.5 h/run**. **Core** matrix: QK{2} × LR{6} × seed{2} ≈ **24 runs** → 2–3 overnight batches.
+- **Stretch** (the depth boundary): add 2 more depths near the threshold (~+18 runs) — or a rented
+  RTX 4090 (~\$0.40/hr) clears it in a few dollars.
+- **Not** GPT-2 scale — an ablation needs *relative* effects, so tiny models are correct, not a compromise.
+
+| Week | Milestone |
+|:--|:--|
+| **5** | Fork nanoGPT · isolated μP-off QK-norm toggle · reproduce baseline · pilot runs |
+| **6** | Core dense LR sweep (≥2 seeds) · log curves |
+| **7** | Analysis + (stretch) depth-threshold runs · figures |
+| **8** | Write Literature Review + Introduction · finalise notebook |
+
+<!-- notes: This fits my laptop: tiny models, about an hour a run, ~24 runs for the core over a couple
+of nights, with the depth boundary as a stretch goal or a few dollars of cloud GPU. Four weeks, one
+milestone each. ~50s -->
+
+---
+
+## 6 · Ethics & Data Access
+
+- **No human subjects, no personal or sensitive data** → no ethical-approval barriers.
+- **Public, openly-licensed datasets** (TinyStories, FineWeb-Edu) — licences to be confirmed and
+  respected; **no data-access constraints.**
+- **Low environmental footprint** — a few GPU-hours total; energy-efficiency is itself a study motivation.
+- **Reproducibility & integrity:** code, configs, seeds, and logs released openly; AI-tool use declared
+  in the research notebook per module policy.
+
+<!-- notes: Ethically low-risk: no people, no private data, only open datasets. The compute footprint is
+tiny — which is part of the point — and everything is released for reproducibility. ~35s -->
+
+---
+
+## 7 · Summary
+
+- **Not "does QK-norm help" (known) — but *when a small model needs it*, charted cleanly.**
+- A **dense, isolated, μP-free** LR sweep at <100M → the **max-stable-LR / LR-sensitivity** benefit *and*
+  the **LR × depth necessity boundary**, **reproducible on a laptop.**
+- An honest **verification-plus-boundary-charting** study: firms up patchy small-scale evidence and
+  hands practitioners a usable rule.
+
+### Thank you — questions welcome
+
+<small>Key references: Henry et al. 2020 · Dehghani et al. 2023 (ViT-22B) · Wortsman et al. (ICLR 2024) ·
+IMU-1 (2026) · OLMo 2 (2025) — *venues/details to verify against primary sources*</small>
+
+<!-- notes: In short: not whether QK-norm helps — we know it can — but pinning down exactly when a small
+model needs it, with a clean, reproducible, laptop-scale experiment that firms up shaky evidence and
+gives builders a usable rule. Thank you. ~30s -->