Add QK-norm toggle, TinyStories prep and automatic run logging
nanoGPT gets a qk_norm flag (off = upstream behaviour), a seed option and a resolved-config dump. run.py logs each run's git commit, config, data checksums, environment and metrics, and stops diverged runs. Claude-Session: https://claude.ai/code/session_01S8KN7NuUCm8nXLmy6eDuVv
This commit is contained in:
1 parent
971caa808c
commit
29e323b2fc
8 files changed
+435
-7
No files matched your search
@@ -0,0 +1,20 @@
|
||||
# nanoGPT (vendored)
|
||||
|
||||
`model.py`, `train.py`, `configurator.py` and `LICENSE` were copied unmodified from
|
||||
[karpathy/nanoGPT](https://github.com/karpathy/nanoGPT) at commit
|
||||
`3adf61e154c3fe3fca428ad6bc3818b27a3b8291` (MIT licence), in their own commit.
|
||||
|
||||
Every change since then is marked `[thesis]` in the code; `git log -p -- 06-experiments/nanogpt`
|
||||
shows them all. Summary:
|
||||
|
||||
| File | Change | Why |
|
||||
|---|---|---|
|
||||
| `model.py` | `qk_norm` config flag; when on, LayerNorm (no bias by default) over each head's queries and keys before the dot product. `1/sqrt(d)` scaling kept | the independent variable. Off = exactly upstream |
|
||||
| `train.py` | `qk_norm`, `seed`, `data_root` config options | QK-norm toggle; seeds for repeat runs (upstream hard-codes 1337); data path |
|
||||
| `train.py` | `configurator.py` loaded relative to the script | so runs can launch from any directory |
|
||||
| `train.py` | writes resolved `config.json` into `out_dir` | run logging |
|
||||
| `train.py` | resume reads `qk_norm` from checkpoint | so resumed runs keep the same architecture |
|
||||
|
||||
**Design choice to confirm:** QK-norm here is LayerNorm-style, as in Dehghani et al. (2023) and
|
||||
Wortsman et al. (2023). Henry et al. (2020) used L2-normalisation with a learnable scale instead,
|
||||
and OLMo 2 uses RMSNorm. `[Your call: which variant the thesis studies.]`
|
||||
Reference in new issue
Block a user