Add QK-norm toggle, TinyStories prep and automatic run logging

nanoGPT gets a qk_norm flag (off = upstream behaviour), a seed option and
a resolved-config dump. run.py logs each run's git commit, config, data
checksums, environment and metrics, and stops diverged runs.

Claude-Session: https://claude.ai/code/session_01S8KN7NuUCm8nXLmy6eDuVv
This commit is contained in:
Cormac Shannon committed 2026-10-28 18:05:31 +00:00
1 parent 971caa808c
commit 29e323b2fc
8 files changed
+435 -7

No files matched your search

+20
View File
@@ -0,0 +1,20 @@
# nanoGPT (vendored)
`model.py`, `train.py`, `configurator.py` and `LICENSE` were copied unmodified from
[karpathy/nanoGPT](https://github.com/karpathy/nanoGPT) at commit
`3adf61e154c3fe3fca428ad6bc3818b27a3b8291` (MIT licence), in their own commit.
Every change since then is marked `[thesis]` in the code; `git log -p -- 06-experiments/nanogpt`
shows them all. Summary:
| File | Change | Why |
|---|---|---|
| `model.py` | `qk_norm` config flag; when on, LayerNorm (no bias by default) over each head's queries and keys before the dot product. `1/sqrt(d)` scaling kept | the independent variable. Off = exactly upstream |
| `train.py` | `qk_norm`, `seed`, `data_root` config options | QK-norm toggle; seeds for repeat runs (upstream hard-codes 1337); data path |
| `train.py` | `configurator.py` loaded relative to the script | so runs can launch from any directory |
| `train.py` | writes resolved `config.json` into `out_dir` | run logging |
| `train.py` | resume reads `qk_norm` from checkpoint | so resumed runs keep the same architecture |
**Design choice to confirm:** QK-norm here is LayerNorm-style, as in Dehghani et al. (2023) and
Wortsman et al. (2023). Henry et al. (2020) used L2-normalisation with a learnable scale instead,
and OLMo 2 uses RMSNorm. `[Your call: which variant the thesis studies.]`