Masters_Thesis/06-experiments

06-experiments

Plain nanoGPT + TinyStories on an M4 Pro (PyTorch MPS), with QK-norm as a single on/off switch and every run logged automatically.

Path	What
preregistration.md	Hypotheses and how each will be tested, written before the main runs
run.py	Launch a run. Writes runs/<id>/run.json + metrics and adds a row to runs/index.csv
configs/	Run configs (spike.py = tech-spike smoke test only)
nanogpt/	Vendored nanoGPT; README-thesis.md lists every change from upstream
data/prepare_tinystories.py	Downloads + tokenises TinyStories, records SHA-256 checksums
runs/	One folder per run (run.json, config.json, metrics.jsonl tracked; checkpoints and raw stdout ignored)
runs/index.csv	One row per run: status, QK-norm, LR, depth, seed, best val loss, diverged-at, commit

Setup (once)

cd 06-experiments
uv venv --python 3.12 .venv && uv pip install --python .venv/bin/python -r requirements.txt
.venv/bin/python data/prepare_tinystories.py --spike   # 22 MB, ~1 min
.venv/bin/python data/prepare_tinystories.py           # full dataset, ~2.2 GB download

Running

.venv/bin/python run.py my-run-name configs/spike.py qk_norm=True learning_rate=3e-3 seed=1 \
    --note "why I'm running this"

Habit: commit before running. run.py records the commit SHA. If there are uncommitted changes, it saves them as diff.patch in the run folder, so a result can always be traced back to the exact code, but a clean commit is easier to cite.

What gets logged (run.json)

name, note, status (completed / diverged / failed / interrupted), start/end time, exact command, requested + resolved config (LR, size, iters, batch, seed, qk_norm…), git commit + dirty flag, upstream nanoGPT commit, dataset name + SHA-256 of raw files + token counts, machine / macOS / Python / torch versions, and a summary (param count, first/last/best loss, iteration where it diverged).

Tech-spike results (28/10/2026)

See runs/index.csv. Pipeline check only: tiny model, 300 steps, small data subset.

Known issues / decisions still open

• Vocabulary size dominates the parameter count. With GPT-2's 50k BPE vocab, a 4-layer 256-wide model has ~16M params, ~13M of them in the embedding. For a "10–15M param" model the non-embedding count is far smaller than it sounds. [Your call: report non-embedding params (as Kaplan et al. and Wortsman et al. do), and/or train a smaller tokenizer on TinyStories.]
• MPS runs in float32 (nanoGPT only autocasts on CUDA). Fine for this study; note it in methods.
• nanoGPT's estimate_mfu assumes an A100, so the mfu number in the logs is meaningless on a Mac.