From 55ba128dcca8310f85e99e172fe920a8c53a3a79 Mon Sep 17 00:00:00 2001 From: Cormac Shannon <> Date: Wed, 28 Oct 2026 18:06:41 +0000 Subject: [PATCH] Add pre-registration draft, timeline skeleton and AI-use log Claude-Session: https://claude.ai/code/session_01S8KN7NuUCm8nXLmy6eDuVv --- 00-supervisor/timeline.md | 38 ++++++++++++++++++++ 06-experiments/README.md | 55 ++++++++++++++++++++++++++++ 06-experiments/preregistration.md | 60 +++++++++++++++++++++++++++++++ AI-USE-LOG.md | 8 +++++ README.md | 14 +++++--- 5 files changed, 170 insertions(+), 5 deletions(-) create mode 100644 00-supervisor/timeline.md create mode 100644 06-experiments/README.md create mode 100644 06-experiments/preregistration.md create mode 100644 AI-USE-LOG.md diff --git a/00-supervisor/timeline.md b/00-supervisor/timeline.md new file mode 100644 index 0000000..ca07fbc --- /dev/null +++ b/00-supervisor/timeline.md @@ -0,0 +1,38 @@ +# Thesis timeline + +*Skeleton, 28/10/2026. Supervisor: "timeline up to me". Interim and final deadlines aren't +known yet, so get them from the handbook / supervisor, then fill in the dates.* + +## Fixed dates (to find out) + +| Milestone | Date | Source | +|---|---|---| +| Interim report / review 1 | `[ ]` | | +| Interim report / review 2 | `[ ]` | | +| Thesis submission | `[ ]` | | +| Viva / presentation | `[ ]` | | + +## Plan + +| Phase | What "done" means | Status | Target | +|---|---|---|---| +| Organise project | `thesis/` repo, git | ✅ 28/10 | | +| Lit: protocol | `02-literature/protocol.md` reviewed and owned | draft ✅ | `[ ]` | +| Lit: survey + snowball round 1 | screening decisions made in `screening.csv` | AI first pass ✅, your decisions pending | next meeting | +| Lit: snowball round 2 | from the `*` includes | not started | `[ ]` | +| Lit: extraction table | full-text read of includes, "how mine differs" for each `*` | not started | `[ ]` | +| Revisit research questions | after extraction | | `[ ]` | +| Tech spike | nanoGPT + TinyStories on MPS, run logging | ✅ 28/10 (smoke tests only) | | +| Pre-registration | `06-experiments/preregistration.md` decisions filled + committed | draft ✅ | `[ ]` before main runs | +| Pilot | time one full-size run; fix per-run budget | | `[ ]` | +| Mechanism logging | attention-logit / entropy / kurtosis hooks | | `[ ]` | +| Main sweep | QK on/off × LR grid × depth × seeds | | `[ ]` | +| Analysis + write-up | | | `[ ]` | + +## Next meeting + +- [ ] survey for lit done (screening decisions) +- [ ] started on snowball (round 1 done; show the flow table in protocol.md section 7) +- [ ] timeline (this file, with dates) +- [ ] tech spike: done early, discuss +- [ ] send anything for him to read **early**, e.g. `protocol.md` diff --git a/06-experiments/README.md b/06-experiments/README.md new file mode 100644 index 0000000..37a5ddd --- /dev/null +++ b/06-experiments/README.md @@ -0,0 +1,55 @@ +# 06-experiments + +Plain nanoGPT + TinyStories on an M4 Pro (PyTorch MPS), with QK-norm as a single on/off switch +and every run logged automatically. + +| Path | What | +|---|---| +| `preregistration.md` | Hypotheses and how each will be tested, written **before** the main runs | +| `run.py` | Launch a run. Writes `runs//run.json` + metrics and adds a row to `runs/index.csv` | +| `configs/` | Run configs (`spike.py` = tech-spike smoke test only) | +| `nanogpt/` | Vendored nanoGPT; `README-thesis.md` lists every change from upstream | +| `data/prepare_tinystories.py` | Downloads + tokenises TinyStories, records SHA-256 checksums | +| `runs/` | One folder per run (`run.json`, `config.json`, `metrics.jsonl` tracked; checkpoints and raw stdout ignored) | +| `runs/index.csv` | One row per run: status, QK-norm, LR, depth, seed, best val loss, diverged-at, commit | + +## Setup (once) + +```sh +cd 06-experiments +uv venv --python 3.12 .venv && uv pip install --python .venv/bin/python -r requirements.txt +.venv/bin/python data/prepare_tinystories.py --spike # 22 MB, ~1 min +.venv/bin/python data/prepare_tinystories.py # full dataset, ~2.2 GB download +``` + +## Running + +```sh +.venv/bin/python run.py my-run-name configs/spike.py qk_norm=True learning_rate=3e-3 seed=1 \ + --note "why I'm running this" +``` + +**Habit:** commit before running. `run.py` records the commit SHA. If there are uncommitted +changes, it saves them as `diff.patch` in the run folder, so a result can always be traced back +to the exact code, but a clean commit is easier to cite. + +## What gets logged (`run.json`) + +name, note, status (`completed` / `diverged` / `failed` / `interrupted`), start/end time, exact +command, requested + resolved config (LR, size, iters, batch, seed, qk_norm…), git commit + +dirty flag, upstream nanoGPT commit, dataset name + SHA-256 of raw files + token counts, +machine / macOS / Python / torch versions, and a summary (param count, first/last/best loss, +iteration where it diverged). + +## Tech-spike results (28/10/2026) + +See `runs/index.csv`. Pipeline check only: tiny model, 300 steps, small data subset. + +## Known issues / decisions still open + +- **Vocabulary size dominates the parameter count.** With GPT-2's 50k BPE vocab, a 4-layer + 256-wide model has ~16M params, ~13M of them in the embedding. For a "10–15M param" model the + non-embedding count is far smaller than it sounds. `[Your call: report non-embedding params + (as Kaplan et al. and Wortsman et al. do), and/or train a smaller tokenizer on TinyStories.]` +- MPS runs in float32 (nanoGPT only autocasts on CUDA). Fine for this study; note it in methods. +- nanoGPT's `estimate_mfu` assumes an A100, so the `mfu` number in the logs is meaningless on a Mac. diff --git a/06-experiments/preregistration.md b/06-experiments/preregistration.md new file mode 100644 index 0000000..78b74eb --- /dev/null +++ b/06-experiments/preregistration.md @@ -0,0 +1,60 @@ +# Pre-registration — QK-norm at <100M parameters + +*Draft v0.1, 28/10/2026. Write the final version **before** the main sweep, commit it, and don't +edit it afterwards: changes go in the amendments table at the bottom, with a date and reason. +The commit timestamp is the proof of when it was registered.* + +Supervisor: "You have a question 1, and then you will have an answer 1 (or a set?) — i.e. does this +happen? If so, does this hold?" Each hypothesis below says what result would support it and what +would count against it, so a surprise is recognisable as a surprise. + +## Research question + +> At <100M parameters, does isolated QK-norm improve training convergence and raise the maximum +> stable learning rate — and at what learning rate does it become necessary? +> Sub-question: how does that threshold shift with model depth? + +*(Wording from the notebook, 21/08/2026. See `02-literature/protocol.md` section 1 for a "how" rewrite to consider.)* + +## Hypotheses (from the notebook, 21/08/2026) + +| | Hypothesis | Supported if… | Not supported if… | +|---|---|---|---| +| H1 | QK-norm's effect is LR-conditional: negligible or slightly negative at low LR, decisive at high LR | `[Your call: e.g. |Δ final val loss| < X at the lowest LRs AND QK-on clearly better or alone stable at the highest LRs]` | `[…]` | +| H2 | QK-norm raises the maximum stable LR and lowers LR-sensitivity | `[e.g. max stable LR(on) > max stable LR(off) across all seeds; LR-sensitivity(on) < (off)]` | `[…]` | +| H3 | The LR at which QK-norm becomes necessary falls as depth increases | `[e.g. the threshold LR decreases monotonically over the depths tested]` | `[…]` | + +## Definitions to fix before running + +These need to be decided up front, because they change the answer. Suggestions follow the +literature in `02-literature/`; the choices are yours. + +| Term | Options / suggestion | Decision | +|---|---|---| +| **Diverged** | loss NaN/inf, or final loss above the initial loss (Wortsman et al. use a final-loss threshold) | `[ ]` | +| **Max stable LR** | largest grid LR where all seeds finish without diverging | `[ ]` | +| **LR-sensitivity** | Wortsman et al. (2023): expected deviation of final loss from the optimum over an LR range, `E_η[min(ℓ(η), ℓ₀) − ℓ*]` — **check the exact formula in the paper** | `[ ]` | +| **Becomes necessary** | lowest LR where QK-off diverges (or is worse by > X) and QK-on doesn't | `[ ]` | +| **Convergence speed** | steps to reach val loss L* | `[ ]` | +| Model sizes / depths | e.g. depth ∈ {2, 4, 8, 12} at fixed width; report non-embedding params | `[ ]` | +| LR grid | e.g. log-spaced, 3e-4 … 3e-1, ~10 points ("dense" vs prior 3-point sweeps) | `[ ]` | +| Seeds | ≥ 2 per cell (notebook says 2); 3 if time allows | `[ ]` | +| Tokens / steps per run | fixed across all cells; constrained by the ~1–1.5 h/run budget | `[ ]` | +| Held fixed | AdamW (β=0.9, 0.95), wd 0.1, warmup, cosine schedule, batch, grad-clip 1.0, μP off, fp32 | `[ ]` | +| Mechanism metrics | max attention logit, attention entropy, activation kurtosis (needs logging hooks, not built yet) | `[ ]` | + +## Analysis plan + +- Plot final val loss vs LR (log axis), QK on/off, one panel per depth, seeds as points. +- Report divergence counts per cell. +- `[Your call: any statistical test, or descriptive only + seed spread.]` + +## Exploratory (not pre-registered) + +Anything noticed during the runs that isn't H1–H3 goes here, clearly labelled as exploratory. + +## Amendments + +| Date | Change | Reason | +|---|---|---| +| | | | diff --git a/AI-USE-LOG.md b/AI-USE-LOG.md new file mode 100644 index 0000000..1441a5c --- /dev/null +++ b/AI-USE-LOG.md @@ -0,0 +1,8 @@ +# GenAI use log + +Record of significant AI-assisted sessions on the thesis, as with the CS6513 notebook's +transparency log. `[Check the thesis handbook's GenAI policy and adjust this format to match.]` + +| Date | Tool | What it did | What I (Cormac) decided / checked | +|---|---|---|---| +| 28/10/2026 | Claude Code (Opus 5.5) | Organised files into `thesis/` + git. Generated `references.bib` from arXiv metadata and found reference errors in the CS6513 paper (`citation-check.md`). Wrote literature-search scripts; ran logged searches (OpenAlex, arXiv) and one snowball round (Semantic Scholar); drafted `protocol.md`; made a first-pass title/abstract screening suggestion (`ai_proposal` column). Set up nanoGPT + TinyStories + run logging (`06-experiments/`); added the `qk_norm` flag; ran 4 smoke-test runs. Drafted `preregistration.md` and `timeline.md` skeletons | `[Your note]` | diff --git a/README.md b/README.md index 6a6734d..cfdc8a6 100644 --- a/README.md +++ b/README.md @@ -3,21 +3,25 @@ **Working title:** When Does a Small Language Model Need QK-Norm? An Isolated Learning-Rate Characterization of Query–Key Normalization at <100M Parameters -Organised copy of the project, made 28/10/2026. Everything here is a **copy** — the -originals are still in the parent folder, and the CS6513 git repo +Organised copy of the project, made 28/10/2026, and tracked in its own git repo (no remote yet). +The originals are still in the parent folder, and the CS6513 git repo (`../Research Project/`, github.com/NikScorch/MSc_Research_Project) is untouched. -This folder is not a git repo yet. + +Not in git (see `.gitignore`): the pitch video (52 MB), `05-course-materials/`, datasets, +checkpoints, the API cache and the AI-assisted `ref_draft`. ## Layout | Folder | What's in it | |---|---| -| `00-supervisor/` | Supervisor meeting notes and to-dos | +| `00-supervisor/` | Supervisor meeting notes and to-dos, `timeline.md` | | `01-proposal/` | CS6513 proposal pitch: Marp slides (`pitch-slides.md`), script, exported PDF, recorded video | -| `02-literature/` | `qknorm-research-findings.md` — prior-art scan and novelty verdict (24/07/2026) | +| `02-literature/` | Lit review: `protocol.md`, search log, screening sheet, extraction table, `references.bib` — see its README | | `03-cs6513-paper/` | CS6513 Introduction + Lit Review paper (IEEE format), see below | | `04-cs6513-notebook/` | CS6513 research notebook (`.rmd` source + exported PDF) | | `05-course-materials/` | CS6513 lecture slides and set readings (wk1 methods, wk2 lit review incl. Xiao 2017 SLR guidance) | +| `06-experiments/` | nanoGPT + TinyStories, QK-norm toggle, automatic run logging, `preregistration.md` — see its README | +| `AI-USE-LOG.md` | GenAI transparency log | ### `03-cs6513-paper/` — which file is which