61 lines
3.5 KiB
Markdown
61 lines
3.5 KiB
Markdown
# Pre-registration — QK-norm at <100M parameters
|
||
|
||
*Draft v0.1, 28/10/2026. Write the final version **before** the main sweep, commit it, and don't
|
||
edit it afterwards: changes go in the amendments table at the bottom, with a date and reason.
|
||
The commit timestamp is the proof of when it was registered.*
|
||
|
||
Supervisor: "You have a question 1, and then you will have an answer 1 (or a set?) — i.e. does this
|
||
happen? If so, does this hold?" Each hypothesis below says what result would support it and what
|
||
would count against it, so a surprise is recognisable as a surprise.
|
||
|
||
## Research question
|
||
|
||
> At <100M parameters, does isolated QK-norm improve training convergence and raise the maximum
|
||
> stable learning rate — and at what learning rate does it become necessary?
|
||
> Sub-question: how does that threshold shift with model depth?
|
||
|
||
*(Wording from the notebook, 21/08/2026. See `02-literature/protocol.md` section 1 for a "how" rewrite to consider.)*
|
||
|
||
## Hypotheses (from the notebook, 21/08/2026)
|
||
|
||
| | Hypothesis | Supported if… | Not supported if… |
|
||
|---|---|---|---|
|
||
| H1 | QK-norm's effect is LR-conditional: negligible or slightly negative at low LR, decisive at high LR | `[Your call: e.g. |Δ final val loss| < X at the lowest LRs AND QK-on clearly better or alone stable at the highest LRs]` | `[…]` |
|
||
| H2 | QK-norm raises the maximum stable LR and lowers LR-sensitivity | `[e.g. max stable LR(on) > max stable LR(off) across all seeds; LR-sensitivity(on) < (off)]` | `[…]` |
|
||
| H3 | The LR at which QK-norm becomes necessary falls as depth increases | `[e.g. the threshold LR decreases monotonically over the depths tested]` | `[…]` |
|
||
|
||
## Definitions to fix before running
|
||
|
||
These need to be decided up front, because they change the answer. Suggestions follow the
|
||
literature in `02-literature/`; the choices are yours.
|
||
|
||
| Term | Options / suggestion | Decision |
|
||
|---|---|---|
|
||
| **Diverged** | loss NaN/inf, or final loss above the initial loss (Wortsman et al. use a final-loss threshold) | `[ ]` |
|
||
| **Max stable LR** | largest grid LR where all seeds finish without diverging | `[ ]` |
|
||
| **LR-sensitivity** | Wortsman et al. (2023): expected deviation of final loss from the optimum over an LR range, `E_η[min(ℓ(η), ℓ₀) − ℓ*]` — **check the exact formula in the paper** | `[ ]` |
|
||
| **Becomes necessary** | lowest LR where QK-off diverges (or is worse by > X) and QK-on doesn't | `[ ]` |
|
||
| **Convergence speed** | steps to reach val loss L* | `[ ]` |
|
||
| Model sizes / depths | e.g. depth ∈ {2, 4, 8, 12} at fixed width; report non-embedding params | `[ ]` |
|
||
| LR grid | e.g. log-spaced, 3e-4 … 3e-1, ~10 points ("dense" vs prior 3-point sweeps) | `[ ]` |
|
||
| Seeds | ≥ 2 per cell (notebook says 2); 3 if time allows | `[ ]` |
|
||
| Tokens / steps per run | fixed across all cells; constrained by the ~1–1.5 h/run budget | `[ ]` |
|
||
| Held fixed | AdamW (β=0.9, 0.95), wd 0.1, warmup, cosine schedule, batch, grad-clip 1.0, μP off, fp32 | `[ ]` |
|
||
| Mechanism metrics | max attention logit, attention entropy, activation kurtosis (needs logging hooks, not built yet) | `[ ]` |
|
||
|
||
## Analysis plan
|
||
|
||
- Plot final val loss vs LR (log axis), QK on/off, one panel per depth, seeds as points.
|
||
- Report divergence counts per cell.
|
||
- `[Your call: any statistical test, or descriptive only + seed spread.]`
|
||
|
||
## Exploratory (not pre-registered)
|
||
|
||
Anything noticed during the runs that isn't H1–H3 goes here, clearly labelled as exploratory.
|
||
|
||
## Amendments
|
||
|
||
| Date | Change | Reason |
|
||
|---|---|---|
|
||
| | | |
|