Cormac Shannon
29e323b2fc
nanoGPT gets a qk_norm flag (off = upstream behaviour), a seed option and a resolved-config dump. run.py logs each run's git commit, config, data checksums, environment and metrics, and stops diverged runs. Claude-Session: https://claude.ai/code/session_01S8KN7NuUCm8nXLmy6eDuVv
1.3 KiB
1.3 KiB
nanoGPT (vendored)
model.py, train.py, configurator.py and LICENSE were copied unmodified from
karpathy/nanoGPT at commit
3adf61e154c3fe3fca428ad6bc3818b27a3b8291 (MIT licence), in their own commit.
Every change since then is marked [thesis] in the code; git log -p -- 06-experiments/nanogpt
shows them all. Summary:
| File | Change | Why |
|---|---|---|
model.py |
qk_norm config flag; when on, LayerNorm (no bias by default) over each head's queries and keys before the dot product. 1/sqrt(d) scaling kept |
the independent variable. Off = exactly upstream |
train.py |
qk_norm, seed, data_root config options |
QK-norm toggle; seeds for repeat runs (upstream hard-codes 1337); data path |
train.py |
configurator.py loaded relative to the script |
so runs can launch from any directory |
train.py |
writes resolved config.json into out_dir |
run logging |
train.py |
resume reads qk_norm from checkpoint |
so resumed runs keep the same architecture |
Design choice to confirm: QK-norm here is LayerNorm-style, as in Dehghani et al. (2023) and
Wortsman et al. (2023). Henry et al. (2020) used L2-normalisation with a learnable scale instead,
and OLMo 2 uses RMSNorm. [Your call: which variant the thesis studies.]