# Pre-registration — QK-norm at <100M parameters

*Draft v0.1, 28/10/2026. Write the final version **before** the main sweep, commit it, and don't
edit it afterwards: changes go in the amendments table at the bottom, with a date and reason.
The commit timestamp is the proof of when it was registered.*

Supervisor: "You have a question 1, and then you will have an answer 1 (or a set?) — i.e. does this
happen? If so, does this hold?" Each hypothesis below says what result would support it and what
would count against it, so a surprise is recognisable as a surprise.

## Research question

> At <100M parameters, does isolated QK-norm improve training convergence and raise the maximum
> stable learning rate — and at what learning rate does it become necessary?
> Sub-question: how does that threshold shift with model depth?

*(Wording from the notebook, 21/08/2026. See `02-literature/protocol.md` section 1 for a "how" rewrite to consider.)*

## Hypotheses (from the notebook, 21/08/2026)

| | Hypothesis | Supported if… | Not supported if… |
|---|---|---|---|
| H1 | QK-norm's effect is LR-conditional: negligible or slightly negative at low LR, decisive at high LR | `[Your call: e.g. |Δ final val loss| < X at the lowest LRs AND QK-on clearly better or alone stable at the highest LRs]` | `[…]` |
| H2 | QK-norm raises the maximum stable LR and lowers LR-sensitivity | `[e.g. max stable LR(on) > max stable LR(off) across all seeds; LR-sensitivity(on) < (off)]` | `[…]` |
| H3 | The LR at which QK-norm becomes necessary falls as depth increases | `[e.g. the threshold LR decreases monotonically over the depths tested]` | `[…]` |

## Definitions to fix before running

These need to be decided up front, because they change the answer. Suggestions follow the
literature in `02-literature/`; the choices are yours.

| Term | Options / suggestion | Decision |
|---|---|---|
| **Diverged** | loss NaN/inf, or final loss above the initial loss (Wortsman et al. use a final-loss threshold) | `[ ]` |
| **Max stable LR** | largest grid LR where all seeds finish without diverging | `[ ]` |
| **LR-sensitivity** | Wortsman et al. (2023): expected deviation of final loss from the optimum over an LR range, `E_η[min(ℓ(η), ℓ₀) − ℓ*]` — **check the exact formula in the paper** | `[ ]` |
| **Becomes necessary** | lowest LR where QK-off diverges (or is worse by > X) and QK-on doesn't | `[ ]` |
| **Convergence speed** | steps to reach val loss L* | `[ ]` |
| Model sizes / depths | e.g. depth ∈ {2, 4, 8, 12} at fixed width; report non-embedding params | `[ ]` |
| LR grid | e.g. log-spaced, 3e-4 … 3e-1, ~10 points ("dense" vs prior 3-point sweeps) | `[ ]` |
| Seeds | ≥ 2 per cell (notebook says 2); 3 if time allows | `[ ]` |
| Tokens / steps per run | fixed across all cells; constrained by the ~1–1.5 h/run budget | `[ ]` |
| Held fixed | AdamW (β=0.9, 0.95), wd 0.1, warmup, cosine schedule, batch, grad-clip 1.0, μP off, fp32 | `[ ]` |
| Mechanism metrics | max attention logit, attention entropy, activation kurtosis (needs logging hooks, not built yet) | `[ ]` |

## Analysis plan

- Plot final val loss vs LR (log axis), QK on/off, one panel per depth, seeds as points.
- Report divergence counts per cell.
- `[Your call: any statistical test, or descriptive only + seed spread.]`

## Exploratory (not pre-registered)

Anything noticed during the runs that isn't H1–H3 goes here, clearly labelled as exploratory.

## Amendments

| Date | Change | Reason |
|---|---|---|
| | | |