3.5 KiB
Pre-registration — QK-norm at <100M parameters
Draft v0.1, 28/10/2026. Write the final version before the main sweep, commit it, and don't edit it afterwards: changes go in the amendments table at the bottom, with a date and reason. The commit timestamp is the proof of when it was registered.
Supervisor: "You have a question 1, and then you will have an answer 1 (or a set?) — i.e. does this happen? If so, does this hold?" Each hypothesis below says what result would support it and what would count against it, so a surprise is recognisable as a surprise.
Research question
At <100M parameters, does isolated QK-norm improve training convergence and raise the maximum stable learning rate — and at what learning rate does it become necessary? Sub-question: how does that threshold shift with model depth?
(Wording from the notebook, 21/08/2026. See 02-literature/protocol.md section 1 for a "how" rewrite to consider.)
Hypotheses (from the notebook, 21/08/2026)
| Hypothesis | Supported if… | Not supported if… | |
|---|---|---|---|
| H1 | QK-norm's effect is LR-conditional: negligible or slightly negative at low LR, decisive at high LR | `[Your call: e.g. | Δ final val loss |
| H2 | QK-norm raises the maximum stable LR and lowers LR-sensitivity | [e.g. max stable LR(on) > max stable LR(off) across all seeds; LR-sensitivity(on) < (off)] |
[…] |
| H3 | The LR at which QK-norm becomes necessary falls as depth increases | [e.g. the threshold LR decreases monotonically over the depths tested] |
[…] |
Definitions to fix before running
These need to be decided up front, because they change the answer. Suggestions follow the
literature in 02-literature/; the choices are yours.
| Term | Options / suggestion | Decision |
|---|---|---|
| Diverged | loss NaN/inf, or final loss above the initial loss (Wortsman et al. use a final-loss threshold) | [ ] |
| Max stable LR | largest grid LR where all seeds finish without diverging | [ ] |
| LR-sensitivity | Wortsman et al. (2023): expected deviation of final loss from the optimum over an LR range, E_η[min(ℓ(η), ℓ₀) − ℓ*] — check the exact formula in the paper |
[ ] |
| Becomes necessary | lowest LR where QK-off diverges (or is worse by > X) and QK-on doesn't | [ ] |
| Convergence speed | steps to reach val loss L* | [ ] |
| Model sizes / depths | e.g. depth ∈ {2, 4, 8, 12} at fixed width; report non-embedding params | [ ] |
| LR grid | e.g. log-spaced, 3e-4 … 3e-1, ~10 points ("dense" vs prior 3-point sweeps) | [ ] |
| Seeds | ≥ 2 per cell (notebook says 2); 3 if time allows | [ ] |
| Tokens / steps per run | fixed across all cells; constrained by the ~1–1.5 h/run budget | [ ] |
| Held fixed | AdamW (β=0.9, 0.95), wd 0.1, warmup, cosine schedule, batch, grad-clip 1.0, μP off, fp32 | [ ] |
| Mechanism metrics | max attention logit, attention entropy, activation kurtosis (needs logging hooks, not built yet) | [ ] |
Analysis plan
- Plot final val loss vs LR (log axis), QK on/off, one panel per depth, seeds as points.
- Report divergence counts per cell.
[Your call: any statistical test, or descriptive only + seed spread.]
Exploratory (not pre-registered)
Anything noticed during the runs that isn't H1–H3 goes here, clearly labelled as exploratory.
Amendments
| Date | Change | Reason |
|---|---|---|