Cormac Shannon
29e323b2fc
nanoGPT gets a qk_norm flag (off = upstream behaviour), a seed option and a resolved-config dump. run.py logs each run's git commit, config, data checksums, environment and metrics, and stops diverged runs. Claude-Session: https://claude.ai/code/session_01S8KN7NuUCm8nXLmy6eDuVv
21 lines
1.3 KiB
Markdown
21 lines
1.3 KiB
Markdown
# nanoGPT (vendored)
|
|
|
|
`model.py`, `train.py`, `configurator.py` and `LICENSE` were copied unmodified from
|
|
[karpathy/nanoGPT](https://github.com/karpathy/nanoGPT) at commit
|
|
`3adf61e154c3fe3fca428ad6bc3818b27a3b8291` (MIT licence), in their own commit.
|
|
|
|
Every change since then is marked `[thesis]` in the code; `git log -p -- 06-experiments/nanogpt`
|
|
shows them all. Summary:
|
|
|
|
| File | Change | Why |
|
|
|---|---|---|
|
|
| `model.py` | `qk_norm` config flag; when on, LayerNorm (no bias by default) over each head's queries and keys before the dot product. `1/sqrt(d)` scaling kept | the independent variable. Off = exactly upstream |
|
|
| `train.py` | `qk_norm`, `seed`, `data_root` config options | QK-norm toggle; seeds for repeat runs (upstream hard-codes 1337); data path |
|
|
| `train.py` | `configurator.py` loaded relative to the script | so runs can launch from any directory |
|
|
| `train.py` | writes resolved `config.json` into `out_dir` | run logging |
|
|
| `train.py` | resume reads `qk_norm` from checkpoint | so resumed runs keep the same architecture |
|
|
|
|
**Design choice to confirm:** QK-norm here is LayerNorm-style, as in Dehghani et al. (2023) and
|
|
Wortsman et al. (2023). Henry et al. (2020) used L2-normalisation with a learnable scale instead,
|
|
and OLMo 2 uses RMSNorm. `[Your call: which variant the thesis studies.]`
|