# nanoGPT (vendored)

`model.py`, `train.py`, `configurator.py` and `LICENSE` were copied unmodified from
[karpathy/nanoGPT](https://github.com/karpathy/nanoGPT) at commit
`3adf61e154c3fe3fca428ad6bc3818b27a3b8291` (MIT licence), in their own commit.

Every change since then is marked `[thesis]` in the code; `git log -p -- 06-experiments/nanogpt`
shows them all. Summary:

| File | Change | Why |
|---|---|---|
| `model.py` | `qk_norm` config flag; when on, LayerNorm (no bias by default) over each head's queries and keys before the dot product. `1/sqrt(d)` scaling kept | the independent variable. Off = exactly upstream |
| `train.py` | `qk_norm`, `seed`, `data_root` config options | QK-norm toggle; seeds for repeat runs (upstream hard-codes 1337); data path |
| `train.py` | `configurator.py` loaded relative to the script | so runs can launch from any directory |
| `train.py` | writes resolved `config.json` into `out_dir` | run logging |
| `train.py` | resume reads `qk_norm` from checkpoint | so resumed runs keep the same architecture |

**Design choice to confirm:** QK-norm here is LayerNorm-style, as in Dehghani et al. (2023) and
Wortsman et al. (2023). Henry et al. (2020) used L2-normalisation with a learnable scale instead,
and OLMo 2 uses RMSNorm. `[Your call: which variant the thesis studies.]`