Tech spike: four smoke-test runs and a divergence-stop test

Pipeline works on MPS (~105 s per 300-step run), results are reproducible,
and the divergence stop works. Not experimental evidence.

Claude-Session: https://claude.ai/code/session_01S8KN7NuUCm8nXLmy6eDuVv

06-experiments/README.md

@@ -43,7 +43,25 @@ iteration where it diverged).
 
 ## Tech-spike results (28/10/2026)
 
-See `runs/index.csv`. Pipeline check only: tiny model, 300 steps, small data subset.
+See `runs/index.csv`. Pipeline check only: 4 layers × 256 wide, 300 steps, 4.9M-token
+subset, one seed. **Not evidence for or against any hypothesis.**
+
+| Run | QK-norm | LR | Best val loss | Time |
+|---|---|---|---|---|
+| spike-qkoff-lr1e-3 | off | 1e-3 | 3.459 | 103 s |
+| spike-qkon-lr1e-3 | on | 1e-3 | 3.430 | 105 s |
+| spike-qkoff-lr1e-1 | off | 1e-1 | 5.295 | 103 s |
+| spike-qkon-lr1e-1 | on | 1e-1 | 3.903 | 106 s |
+| test-divergence-stop | off | 1e-1 | (stopped) | 6 s |
+
+What the spike showed:
+- Training runs on MPS at ~0.29 s/step for this size; the whole pipeline works.
+- **Reproducible:** re-running QK-off at 1e-3 gave the identical val loss (3.4590) as a pre-commit test run.
+- At LR 0.1, QK-off was clearly worse but **did not diverge**: grad clipping (1.0) kept loss finite.
+  So "diverged" needs a definition beyond NaN for the pre-registration (e.g. final loss vs a threshold).
+- The divergence stop works (tested with an artificially low threshold; the run was killed and logged).
+- The QK-on vs off direction at high LR matches H2, but this is one seed, 300 steps: a hint for the
+  pilot, nothing more.
 
 ## Known issues / decisions still open
 

06-experiments/runs/20261028-180656_spike-qkoff-lr1e-3/config.json

@@ -0,0 +1,38 @@
+{
+  "out_dir": "/Users/nik/ISE/9.CS6513/thesis/06-experiments/runs/20261028-180656_spike-qkoff-lr1e-3",
+  "eval_interval": 100,
+  "log_interval": 10,
+  "eval_iters": 20,
+  "eval_only": false,
+  "always_save_checkpoint": false,
+  "init_from": "scratch",
+  "wandb_log": false,
+  "wandb_project": "owt",
+  "wandb_run_name": "gpt2",
+  "dataset": "tinystories_spike",
+  "gradient_accumulation_steps": 1,
+  "batch_size": 32,
+  "block_size": 256,
+  "n_layer": 4,
+  "n_head": 4,
+  "n_embd": 256,
+  "dropout": 0.0,
+  "bias": false,
+  "qk_norm": false,
+  "seed": 1337,
+  "data_root": "/Users/nik/ISE/9.CS6513/thesis/06-experiments/data",
+  "learning_rate": 0.001,
+  "max_iters": 300,
+  "weight_decay": 0.1,
+  "beta1": 0.9,
+  "beta2": 0.95,
+  "grad_clip": 1.0,
+  "decay_lr": true,
+  "warmup_iters": 30,
+  "lr_decay_iters": 300,
+  "min_lr": 0.0001,
+  "backend": "nccl",
+  "device": "mps",
+  "dtype": "float32",
+  "compile": false
+}

06-experiments/runs/20261028-180656_spike-qkoff-lr1e-3/metrics.jsonl

@@ -0,0 +1,35 @@
+{"type": "eval", "iter": 0, "train_loss": 10.9141, "val_loss": 10.9167, "t": 5.28}
+{"type": "train", "iter": 0, "loss": 10.9193, "ms": 4306.09, "t": 5.61}
+{"type": "train", "iter": 10, "loss": 9.717, "ms": 277.79, "t": 8.4}
+{"type": "train", "iter": 20, "loss": 7.856, "ms": 278.89, "t": 11.19}
+{"type": "train", "iter": 30, "loss": 5.8282, "ms": 280.65, "t": 13.98}
+{"type": "train", "iter": 40, "loss": 5.188, "ms": 280.42, "t": 16.81}
+{"type": "train", "iter": 50, "loss": 4.7399, "ms": 279.76, "t": 19.6}
+{"type": "train", "iter": 60, "loss": 4.4652, "ms": 279.16, "t": 22.4}
+{"type": "train", "iter": 70, "loss": 4.3047, "ms": 279.57, "t": 25.2}
+{"type": "train", "iter": 80, "loss": 4.1141, "ms": 280.67, "t": 28.0}
+{"type": "train", "iter": 90, "loss": 4.0799, "ms": 279.89, "t": 30.8}
+{"type": "eval", "iter": 100, "train_loss": 4.0112, "val_loss": 4.0447, "t": 37.28}
+{"type": "train", "iter": 100, "loss": 3.9923, "ms": 4480.26, "t": 37.81}
+{"type": "train", "iter": 110, "loss": 3.9961, "ms": 280.65, "t": 40.61}
+{"type": "train", "iter": 120, "loss": 3.802, "ms": 280.13, "t": 43.4}
+{"type": "train", "iter": 130, "loss": 3.8455, "ms": 279.15, "t": 46.2}
+{"type": "train", "iter": 140, "loss": 3.8329, "ms": 278.97, "t": 48.99}
+{"type": "train", "iter": 150, "loss": 3.8187, "ms": 279.6, "t": 51.79}
+{"type": "train", "iter": 160, "loss": 3.683, "ms": 280.52, "t": 54.59}
+{"type": "train", "iter": 170, "loss": 3.7639, "ms": 280.91, "t": 57.39}
+{"type": "train", "iter": 180, "loss": 3.6422, "ms": 280.01, "t": 60.19}
+{"type": "train", "iter": 190, "loss": 3.4959, "ms": 294.68, "t": 63.04}
+{"type": "eval", "iter": 200, "train_loss": 3.5542, "val_loss": 3.5984, "t": 69.66}
+{"type": "train", "iter": 200, "loss": 3.4831, "ms": 4487.24, "t": 70.09}
+{"type": "train", "iter": 210, "loss": 3.5612, "ms": 280.52, "t": 72.91}
+{"type": "train", "iter": 220, "loss": 3.5742, "ms": 291.78, "t": 75.76}
+{"type": "train", "iter": 230, "loss": 3.458, "ms": 282.25, "t": 78.58}
+{"type": "train", "iter": 240, "loss": 3.5315, "ms": 281.29, "t": 81.4}
+{"type": "train", "iter": 250, "loss": 3.4362, "ms": 281.5, "t": 84.21}
+{"type": "train", "iter": 260, "loss": 3.4534, "ms": 281.67, "t": 87.03}
+{"type": "train", "iter": 270, "loss": 3.4163, "ms": 281.69, "t": 89.85}
+{"type": "train", "iter": 280, "loss": 3.3884, "ms": 282.78, "t": 92.67}
+{"type": "train", "iter": 290, "loss": 3.3656, "ms": 292.04, "t": 95.52}
+{"type": "eval", "iter": 300, "train_loss": 3.3974, "val_loss": 3.459, "t": 102.13}
+{"type": "train", "iter": 300, "loss": 3.3888, "ms": 4461.83, "t": 102.56}

06-experiments/runs/20261028-180656_spike-qkoff-lr1e-3/run.json

@@ -0,0 +1,86 @@
+{
+  "run_id": "20261028-180656_spike-qkoff-lr1e-3",
+  "name": "spike-qkoff-lr1e-3",
+  "note": "tech spike smoke test (not an experiment)",
+  "status": "completed",
+  "started": "2026-10-28T18:06:56+00:00",
+  "command": "/Users/nik/ISE/9.CS6513/thesis/06-experiments/.venv/bin/python /Users/nik/ISE/9.CS6513/thesis/06-experiments/nanogpt/train.py configs/spike.py --out_dir=/Users/nik/ISE/9.CS6513/thesis/06-experiments/runs/20261028-180656_spike-qkoff-lr1e-3 --data_root=/Users/nik/ISE/9.CS6513/thesis/06-experiments/data --qk_norm=False --learning_rate=1e-3",
+  "config_file": "configs/spike.py",
+  "overrides": [
+    "qk_norm=False",
+    "learning_rate=1e-3"
+  ],
+  "config_requested": {
+    "dataset": "tinystories_spike",
+    "device": "mps",
+    "dtype": "float32",
+    "compile": false,
+    "n_layer": 4,
+    "n_head": 4,
+    "n_embd": 256,
+    "block_size": 256,
+    "bias": false,
+    "dropout": 0.0,
+    "qk_norm": false,
+    "batch_size": 32,
+    "gradient_accumulation_steps": 1,
+    "learning_rate": 0.001,
+    "max_iters": 300,
+    "lr_decay_iters": 300,
+    "min_lr": 0.0001,
+    "warmup_iters": 30,
+    "weight_decay": 0.1,
+    "grad_clip": 1.0,
+    "eval_interval": 100,
+    "eval_iters": 20,
+    "log_interval": 10,
+    "always_save_checkpoint": false,
+    "seed": 1337
+  },
+  "git": {
+    "commit": "55ba128dcca8310f85e99e172fe920a8c53a3a79",
+    "branch": "main",
+    "dirty": false
+  },
+  "nanogpt_upstream_commit": "3adf61e154c3fe3fca428ad6bc3818b27a3b8291",
+  "data": {
+    "dataset": "TinyStories V2 (GPT-4)",
+    "source": "https://huggingface.co/datasets/roneneldan/TinyStories/resolve/main/",
+    "tokenizer": "tiktoken gpt2",
+    "files": {
+      "TinyStoriesV2-GPT4-valid.txt": "6874bae9a4c1a4e7edcf0e53b86c17817e9cf881fc75ff2368da457b80c0585d"
+    },
+    "split": "valid file split 90/10 by story order: 24867 train / 2763 val stories",
+    "tokens": {
+      "train": 4920360,
+      "val": 548664
+    },
+    "train_bin_bytes": 9840720,
+    "val_bin_bytes": 1097328
+  },
+  "environment": {
+    "host": "Cormacs-MacBook-Pro.local",
+    "platform": "macOS-15.7.7-arm64-arm-64bit",
+    "machine": "arm64",
+    "cpu": "Apple M4 Pro",
+    "python": "3.12.14",
+    "torch": "2.15.0",
+    "mps_available": true,
+    "cuda_available": false
+  },
+  "finished": "2026-10-28T18:08:40+00:00",
+  "wall_seconds": 102.9,
+  "summary": {
+    "params_millions_nanogpt_count": 16.03,
+    "first_train_loss": 10.9193,
+    "last_train_loss": 3.3888,
+    "last_eval": {
+      "iter": 300,
+      "train_loss": 3.3974,
+      "val_loss": 3.459
+    },
+    "best_val_loss": 3.459,
+    "diverged_at_iter": null,
+    "returncode": 0
+  }
+}

06-experiments/runs/20261028-180840_spike-qkon-lr1e-3/config.json

@@ -0,0 +1,38 @@
+{
+  "out_dir": "/Users/nik/ISE/9.CS6513/thesis/06-experiments/runs/20261028-180840_spike-qkon-lr1e-3",
+  "eval_interval": 100,
+  "log_interval": 10,
+  "eval_iters": 20,
+  "eval_only": false,
+  "always_save_checkpoint": false,
+  "init_from": "scratch",
+  "wandb_log": false,
+  "wandb_project": "owt",
+  "wandb_run_name": "gpt2",
+  "dataset": "tinystories_spike",
+  "gradient_accumulation_steps": 1,
+  "batch_size": 32,
+  "block_size": 256,
+  "n_layer": 4,
+  "n_head": 4,
+  "n_embd": 256,
+  "dropout": 0.0,
+  "bias": false,
+  "qk_norm": true,
+  "seed": 1337,
+  "data_root": "/Users/nik/ISE/9.CS6513/thesis/06-experiments/data",
+  "learning_rate": 0.001,
+  "max_iters": 300,
+  "weight_decay": 0.1,
+  "beta1": 0.9,
+  "beta2": 0.95,
+  "grad_clip": 1.0,
+  "decay_lr": true,
+  "warmup_iters": 30,
+  "lr_decay_iters": 300,
+  "min_lr": 0.0001,
+  "backend": "nccl",
+  "device": "mps",
+  "dtype": "float32",
+  "compile": false
+}

06-experiments/runs/20261028-180840_spike-qkon-lr1e-3/metrics.jsonl

@@ -0,0 +1,35 @@
+{"type": "eval", "iter": 0, "train_loss": 10.9101, "val_loss": 10.9119, "t": 5.5}
+{"type": "train", "iter": 0, "loss": 10.9141, "ms": 4672.21, "t": 5.94}
+{"type": "train", "iter": 10, "loss": 9.718, "ms": 287.96, "t": 8.8}
+{"type": "train", "iter": 20, "loss": 7.8391, "ms": 284.7, "t": 11.65}
+{"type": "train", "iter": 30, "loss": 5.8285, "ms": 284.87, "t": 14.5}
+{"type": "train", "iter": 40, "loss": 5.1912, "ms": 290.48, "t": 17.38}
+{"type": "train", "iter": 50, "loss": 4.7577, "ms": 288.18, "t": 20.27}
+{"type": "train", "iter": 60, "loss": 4.4781, "ms": 285.07, "t": 23.13}
+{"type": "train", "iter": 70, "loss": 4.312, "ms": 285.94, "t": 25.98}
+{"type": "train", "iter": 80, "loss": 4.1143, "ms": 285.6, "t": 28.83}
+{"type": "train", "iter": 90, "loss": 4.0708, "ms": 285.39, "t": 31.68}
+{"type": "eval", "iter": 100, "train_loss": 3.9981, "val_loss": 4.0336, "t": 38.37}
+{"type": "train", "iter": 100, "loss": 3.9857, "ms": 4657.44, "t": 38.93}
+{"type": "train", "iter": 110, "loss": 3.9788, "ms": 286.19, "t": 41.8}
+{"type": "train", "iter": 120, "loss": 3.7863, "ms": 284.34, "t": 44.65}
+{"type": "train", "iter": 130, "loss": 3.8257, "ms": 286.31, "t": 47.5}
+{"type": "train", "iter": 140, "loss": 3.8187, "ms": 286.03, "t": 50.36}
+{"type": "train", "iter": 150, "loss": 3.7967, "ms": 284.16, "t": 53.21}
+{"type": "train", "iter": 160, "loss": 3.657, "ms": 285.59, "t": 56.07}
+{"type": "train", "iter": 170, "loss": 3.735, "ms": 283.65, "t": 58.92}
+{"type": "train", "iter": 180, "loss": 3.6052, "ms": 288.55, "t": 61.78}
+{"type": "train", "iter": 190, "loss": 3.4672, "ms": 284.84, "t": 64.64}
+{"type": "eval", "iter": 200, "train_loss": 3.5267, "val_loss": 3.569, "t": 71.33}
+{"type": "train", "iter": 200, "loss": 3.4533, "ms": 4560.67, "t": 71.78}
+{"type": "train", "iter": 210, "loss": 3.5281, "ms": 284.63, "t": 74.63}
+{"type": "train", "iter": 220, "loss": 3.5519, "ms": 287.88, "t": 77.48}
+{"type": "train", "iter": 230, "loss": 3.435, "ms": 284.81, "t": 80.32}
+{"type": "train", "iter": 240, "loss": 3.4945, "ms": 284.65, "t": 83.17}
+{"type": "train", "iter": 250, "loss": 3.4147, "ms": 284.48, "t": 86.02}
+{"type": "train", "iter": 260, "loss": 3.422, "ms": 284.64, "t": 88.87}
+{"type": "train", "iter": 270, "loss": 3.3949, "ms": 283.83, "t": 91.72}
+{"type": "train", "iter": 280, "loss": 3.362, "ms": 284.39, "t": 94.58}
+{"type": "train", "iter": 290, "loss": 3.3412, "ms": 284.93, "t": 97.46}
+{"type": "eval", "iter": 300, "train_loss": 3.3683, "val_loss": 3.4304, "t": 104.06}
+{"type": "train", "iter": 300, "loss": 3.3557, "ms": 4477.21, "t": 104.51}

06-experiments/runs/20261028-180840_spike-qkon-lr1e-3/run.json

@@ -0,0 +1,86 @@
+{
+  "run_id": "20261028-180840_spike-qkon-lr1e-3",
+  "name": "spike-qkon-lr1e-3",
+  "note": "tech spike smoke test (not an experiment)",
+  "status": "completed",
+  "started": "2026-10-28T18:08:40+00:00",
+  "command": "/Users/nik/ISE/9.CS6513/thesis/06-experiments/.venv/bin/python /Users/nik/ISE/9.CS6513/thesis/06-experiments/nanogpt/train.py configs/spike.py --out_dir=/Users/nik/ISE/9.CS6513/thesis/06-experiments/runs/20261028-180840_spike-qkon-lr1e-3 --data_root=/Users/nik/ISE/9.CS6513/thesis/06-experiments/data --qk_norm=True --learning_rate=1e-3",
+  "config_file": "configs/spike.py",
+  "overrides": [
+    "qk_norm=True",
+    "learning_rate=1e-3"
+  ],
+  "config_requested": {
+    "dataset": "tinystories_spike",
+    "device": "mps",
+    "dtype": "float32",
+    "compile": false,
+    "n_layer": 4,
+    "n_head": 4,
+    "n_embd": 256,
+    "block_size": 256,
+    "bias": false,
+    "dropout": 0.0,
+    "qk_norm": true,
+    "batch_size": 32,
+    "gradient_accumulation_steps": 1,
+    "learning_rate": 0.001,
+    "max_iters": 300,
+    "lr_decay_iters": 300,
+    "min_lr": 0.0001,
+    "warmup_iters": 30,
+    "weight_decay": 0.1,
+    "grad_clip": 1.0,
+    "eval_interval": 100,
+    "eval_iters": 20,
+    "log_interval": 10,
+    "always_save_checkpoint": false,
+    "seed": 1337
+  },
+  "git": {
+    "commit": "55ba128dcca8310f85e99e172fe920a8c53a3a79",
+    "branch": "main",
+    "dirty": false
+  },
+  "nanogpt_upstream_commit": "3adf61e154c3fe3fca428ad6bc3818b27a3b8291",
+  "data": {
+    "dataset": "TinyStories V2 (GPT-4)",
+    "source": "https://huggingface.co/datasets/roneneldan/TinyStories/resolve/main/",
+    "tokenizer": "tiktoken gpt2",
+    "files": {
+      "TinyStoriesV2-GPT4-valid.txt": "6874bae9a4c1a4e7edcf0e53b86c17817e9cf881fc75ff2368da457b80c0585d"
+    },
+    "split": "valid file split 90/10 by story order: 24867 train / 2763 val stories",
+    "tokens": {
+      "train": 4920360,
+      "val": 548664
+    },
+    "train_bin_bytes": 9840720,
+    "val_bin_bytes": 1097328
+  },
+  "environment": {
+    "host": "Cormacs-MacBook-Pro.local",
+    "platform": "macOS-15.7.7-arm64-arm-64bit",
+    "machine": "arm64",
+    "cpu": "Apple M4 Pro",
+    "python": "3.12.14",
+    "torch": "2.15.0",
+    "mps_available": true,
+    "cuda_available": false
+  },
+  "finished": "2026-10-28T18:10:25+00:00",
+  "wall_seconds": 104.8,
+  "summary": {
+    "params_millions_nanogpt_count": 16.03,
+    "first_train_loss": 10.9141,
+    "last_train_loss": 3.3557,
+    "last_eval": {
+      "iter": 300,
+      "train_loss": 3.3683,
+      "val_loss": 3.4304
+    },
+    "best_val_loss": 3.4304,
+    "diverged_at_iter": null,
+    "returncode": 0
+  }
+}

06-experiments/runs/20261028-181025_spike-qkoff-lr1e-1/config.json

@@ -0,0 +1,38 @@
+{
+  "out_dir": "/Users/nik/ISE/9.CS6513/thesis/06-experiments/runs/20261028-181025_spike-qkoff-lr1e-1",
+  "eval_interval": 100,
+  "log_interval": 10,
+  "eval_iters": 20,
+  "eval_only": false,
+  "always_save_checkpoint": false,
+  "init_from": "scratch",
+  "wandb_log": false,
+  "wandb_project": "owt",
+  "wandb_run_name": "gpt2",
+  "dataset": "tinystories_spike",
+  "gradient_accumulation_steps": 1,
+  "batch_size": 32,
+  "block_size": 256,
+  "n_layer": 4,
+  "n_head": 4,
+  "n_embd": 256,
+  "dropout": 0.0,
+  "bias": false,
+  "qk_norm": false,
+  "seed": 1337,
+  "data_root": "/Users/nik/ISE/9.CS6513/thesis/06-experiments/data",
+  "learning_rate": 0.1,
+  "max_iters": 300,
+  "weight_decay": 0.1,
+  "beta1": 0.9,
+  "beta2": 0.95,
+  "grad_clip": 1.0,
+  "decay_lr": true,
+  "warmup_iters": 30,
+  "lr_decay_iters": 300,
+  "min_lr": 0.01,
+  "backend": "nccl",
+  "device": "mps",
+  "dtype": "float32",
+  "compile": false
+}

06-experiments/runs/20261028-181025_spike-qkoff-lr1e-1/metrics.jsonl

@@ -0,0 +1,35 @@
+{"type": "eval", "iter": 0, "train_loss": 10.9141, "val_loss": 10.9167, "t": 5.41}
+{"type": "train", "iter": 0, "loss": 10.9193, "ms": 4331.0, "t": 5.74}
+{"type": "train", "iter": 10, "loss": 7.6884, "ms": 279.87, "t": 8.54}
+{"type": "train", "iter": 20, "loss": 6.7757, "ms": 280.21, "t": 11.34}
+{"type": "train", "iter": 30, "loss": 5.8674, "ms": 279.61, "t": 14.14}
+{"type": "train", "iter": 40, "loss": 5.9878, "ms": 280.23, "t": 16.95}
+{"type": "train", "iter": 50, "loss": 5.9047, "ms": 280.35, "t": 19.75}
+{"type": "train", "iter": 60, "loss": 5.8552, "ms": 280.38, "t": 22.55}
+{"type": "train", "iter": 70, "loss": 5.8407, "ms": 279.0, "t": 25.36}
+{"type": "train", "iter": 80, "loss": 5.7461, "ms": 280.14, "t": 28.16}
+{"type": "train", "iter": 90, "loss": 5.7984, "ms": 280.12, "t": 30.96}
+{"type": "eval", "iter": 100, "train_loss": 5.7546, "val_loss": 5.789, "t": 37.47}
+{"type": "train", "iter": 100, "loss": 5.7697, "ms": 4495.39, "t": 37.99}
+{"type": "train", "iter": 110, "loss": 5.7763, "ms": 280.61, "t": 40.79}
+{"type": "train", "iter": 120, "loss": 5.6788, "ms": 279.57, "t": 43.6}
+{"type": "train", "iter": 130, "loss": 5.7067, "ms": 292.17, "t": 46.47}
+{"type": "train", "iter": 140, "loss": 5.6992, "ms": 281.82, "t": 49.28}
+{"type": "train", "iter": 150, "loss": 5.6794, "ms": 292.66, "t": 52.14}
+{"type": "train", "iter": 160, "loss": 5.6159, "ms": 295.05, "t": 55.09}
+{"type": "train", "iter": 170, "loss": 5.6578, "ms": 280.39, "t": 57.92}
+{"type": "train", "iter": 180, "loss": 5.5731, "ms": 281.59, "t": 60.74}
+{"type": "train", "iter": 190, "loss": 5.3993, "ms": 298.74, "t": 63.59}
+{"type": "eval", "iter": 200, "train_loss": 5.4974, "val_loss": 5.5322, "t": 70.3}
+{"type": "train", "iter": 200, "loss": 5.4276, "ms": 4449.02, "t": 70.73}
+{"type": "train", "iter": 210, "loss": 5.524, "ms": 290.66, "t": 73.6}
+{"type": "train", "iter": 220, "loss": 5.4767, "ms": 281.67, "t": 76.42}
+{"type": "train", "iter": 230, "loss": 5.485, "ms": 282.2, "t": 79.23}
+{"type": "train", "iter": 240, "loss": 5.4442, "ms": 281.09, "t": 82.04}
+{"type": "train", "iter": 250, "loss": 5.3695, "ms": 281.49, "t": 84.86}
+{"type": "train", "iter": 260, "loss": 5.3708, "ms": 281.08, "t": 87.67}
+{"type": "train", "iter": 270, "loss": 5.36, "ms": 282.22, "t": 90.48}
+{"type": "train", "iter": 280, "loss": 5.2385, "ms": 281.0, "t": 93.29}
+{"type": "train", "iter": 290, "loss": 5.2615, "ms": 280.38, "t": 96.1}
+{"type": "eval", "iter": 300, "train_loss": 5.2472, "val_loss": 5.2948, "t": 102.62}
+{"type": "train", "iter": 300, "loss": 5.2321, "ms": 4418.66, "t": 103.06}

06-experiments/runs/20261028-181025_spike-qkoff-lr1e-1/run.json

@@ -0,0 +1,87 @@
+{
+  "run_id": "20261028-181025_spike-qkoff-lr1e-1",
+  "name": "spike-qkoff-lr1e-1",
+  "note": "tech spike smoke test (not an experiment); divergence-detection check",
+  "status": "completed",
+  "started": "2026-10-28T18:10:25+00:00",
+  "command": "/Users/nik/ISE/9.CS6513/thesis/06-experiments/.venv/bin/python /Users/nik/ISE/9.CS6513/thesis/06-experiments/nanogpt/train.py configs/spike.py --out_dir=/Users/nik/ISE/9.CS6513/thesis/06-experiments/runs/20261028-181025_spike-qkoff-lr1e-1 --data_root=/Users/nik/ISE/9.CS6513/thesis/06-experiments/data --qk_norm=False --learning_rate=1e-1 --min_lr=1e-2",
+  "config_file": "configs/spike.py",
+  "overrides": [
+    "qk_norm=False",
+    "learning_rate=1e-1",
+    "min_lr=1e-2"
+  ],
+  "config_requested": {
+    "dataset": "tinystories_spike",
+    "device": "mps",
+    "dtype": "float32",
+    "compile": false,
+    "n_layer": 4,
+    "n_head": 4,
+    "n_embd": 256,
+    "block_size": 256,
+    "bias": false,
+    "dropout": 0.0,
+    "qk_norm": false,
+    "batch_size": 32,
+    "gradient_accumulation_steps": 1,
+    "learning_rate": 0.1,
+    "max_iters": 300,
+    "lr_decay_iters": 300,
+    "min_lr": 0.01,
+    "warmup_iters": 30,
+    "weight_decay": 0.1,
+    "grad_clip": 1.0,
+    "eval_interval": 100,
+    "eval_iters": 20,
+    "log_interval": 10,
+    "always_save_checkpoint": false,
+    "seed": 1337
+  },
+  "git": {
+    "commit": "55ba128dcca8310f85e99e172fe920a8c53a3a79",
+    "branch": "main",
+    "dirty": false
+  },
+  "nanogpt_upstream_commit": "3adf61e154c3fe3fca428ad6bc3818b27a3b8291",
+  "data": {
+    "dataset": "TinyStories V2 (GPT-4)",
+    "source": "https://huggingface.co/datasets/roneneldan/TinyStories/resolve/main/",
+    "tokenizer": "tiktoken gpt2",
+    "files": {
+      "TinyStoriesV2-GPT4-valid.txt": "6874bae9a4c1a4e7edcf0e53b86c17817e9cf881fc75ff2368da457b80c0585d"
+    },
+    "split": "valid file split 90/10 by story order: 24867 train / 2763 val stories",
+    "tokens": {
+      "train": 4920360,
+      "val": 548664
+    },
+    "train_bin_bytes": 9840720,
+    "val_bin_bytes": 1097328
+  },
+  "environment": {
+    "host": "Cormacs-MacBook-Pro.local",
+    "platform": "macOS-15.7.7-arm64-arm-64bit",
+    "machine": "arm64",
+    "cpu": "Apple M4 Pro",
+    "python": "3.12.14",
+    "torch": "2.15.0",
+    "mps_available": true,
+    "cuda_available": false
+  },
+  "finished": "2026-10-28T18:12:09+00:00",
+  "wall_seconds": 103.4,
+  "summary": {
+    "params_millions_nanogpt_count": 16.03,
+    "first_train_loss": 10.9193,
+    "last_train_loss": 5.2321,
+    "last_eval": {
+      "iter": 300,
+      "train_loss": 5.2472,
+      "val_loss": 5.2948
+    },
+    "best_val_loss": 5.2948,
+    "diverged_at_iter": null,
+    "returncode": 0
+  }
+}

06-experiments/runs/20261028-181209_spike-qkon-lr1e-1/config.json

@@ -0,0 +1,38 @@
+{
+  "out_dir": "/Users/nik/ISE/9.CS6513/thesis/06-experiments/runs/20261028-181209_spike-qkon-lr1e-1",
+  "eval_interval": 100,
+  "log_interval": 10,
+  "eval_iters": 20,
+  "eval_only": false,
+  "always_save_checkpoint": false,
+  "init_from": "scratch",
+  "wandb_log": false,
+  "wandb_project": "owt",
+  "wandb_run_name": "gpt2",
+  "dataset": "tinystories_spike",
+  "gradient_accumulation_steps": 1,
+  "batch_size": 32,
+  "block_size": 256,
+  "n_layer": 4,
+  "n_head": 4,
+  "n_embd": 256,
+  "dropout": 0.0,
+  "bias": false,
+  "qk_norm": true,
+  "seed": 1337,
+  "data_root": "/Users/nik/ISE/9.CS6513/thesis/06-experiments/data",
+  "learning_rate": 0.1,
+  "max_iters": 300,
+  "weight_decay": 0.1,
+  "beta1": 0.9,
+  "beta2": 0.95,
+  "grad_clip": 1.0,
+  "decay_lr": true,
+  "warmup_iters": 30,
+  "lr_decay_iters": 300,
+  "min_lr": 0.01,
+  "backend": "nccl",
+  "device": "mps",
+  "dtype": "float32",
+  "compile": false
+}

06-experiments/runs/20261028-181209_spike-qkon-lr1e-1/metrics.jsonl

@@ -0,0 +1,35 @@
+{"type": "eval", "iter": 0, "train_loss": 10.9101, "val_loss": 10.9119, "t": 5.35}
+{"type": "train", "iter": 0, "loss": 10.9141, "ms": 4381.39, "t": 5.68}
+{"type": "train", "iter": 10, "loss": 7.244, "ms": 282.91, "t": 8.5}
+{"type": "train", "iter": 20, "loss": 6.685, "ms": 282.55, "t": 11.33}
+{"type": "train", "iter": 30, "loss": 5.8816, "ms": 282.81, "t": 14.15}
+{"type": "train", "iter": 40, "loss": 6.0125, "ms": 282.89, "t": 16.98}
+{"type": "train", "iter": 50, "loss": 5.9419, "ms": 281.66, "t": 19.81}
+{"type": "train", "iter": 60, "loss": 5.8631, "ms": 283.47, "t": 22.71}
+{"type": "train", "iter": 70, "loss": 5.6184, "ms": 287.64, "t": 25.59}
+{"type": "train", "iter": 80, "loss": 5.3906, "ms": 294.14, "t": 28.56}
+{"type": "train", "iter": 90, "loss": 5.2812, "ms": 284.17, "t": 31.53}
+{"type": "eval", "iter": 100, "train_loss": 5.1683, "val_loss": 5.2052, "t": 38.32}
+{"type": "train", "iter": 100, "loss": 5.1892, "ms": 4687.97, "t": 38.88}
+{"type": "train", "iter": 110, "loss": 5.1302, "ms": 284.39, "t": 41.78}
+{"type": "train", "iter": 120, "loss": 4.8073, "ms": 284.14, "t": 44.63}
+{"type": "train", "iter": 130, "loss": 4.7233, "ms": 284.49, "t": 47.47}
+{"type": "train", "iter": 140, "loss": 4.604, "ms": 284.17, "t": 50.32}
+{"type": "train", "iter": 150, "loss": 4.5495, "ms": 284.19, "t": 53.21}
+{"type": "train", "iter": 160, "loss": 4.376, "ms": 286.09, "t": 56.2}
+{"type": "train", "iter": 170, "loss": 4.469, "ms": 309.41, "t": 59.19}
+{"type": "train", "iter": 180, "loss": 4.3172, "ms": 284.88, "t": 62.19}
+{"type": "train", "iter": 190, "loss": 4.1296, "ms": 294.5, "t": 65.17}
+{"type": "eval", "iter": 200, "train_loss": 4.2006, "val_loss": 4.2389, "t": 71.94}
+{"type": "train", "iter": 200, "loss": 4.1261, "ms": 4612.6, "t": 72.39}
+{"type": "train", "iter": 210, "loss": 4.2153, "ms": 292.29, "t": 75.27}
+{"type": "train", "iter": 220, "loss": 4.1696, "ms": 283.53, "t": 78.13}
+{"type": "train", "iter": 230, "loss": 4.0597, "ms": 290.17, "t": 81.02}
+{"type": "train", "iter": 240, "loss": 4.0813, "ms": 286.98, "t": 83.9}
+{"type": "train", "iter": 250, "loss": 3.9827, "ms": 284.4, "t": 86.76}
+{"type": "train", "iter": 260, "loss": 3.9723, "ms": 284.78, "t": 89.61}
+{"type": "train", "iter": 270, "loss": 3.928, "ms": 287.37, "t": 92.47}
+{"type": "train", "iter": 280, "loss": 3.8621, "ms": 284.67, "t": 95.32}
+{"type": "train", "iter": 290, "loss": 3.8471, "ms": 283.42, "t": 98.17}
+{"type": "eval", "iter": 300, "train_loss": 3.8471, "val_loss": 3.9026, "t": 104.93}
+{"type": "train", "iter": 300, "loss": 3.8235, "ms": 4652.7, "t": 105.39}

06-experiments/runs/20261028-181209_spike-qkon-lr1e-1/run.json

@@ -0,0 +1,87 @@
+{
+  "run_id": "20261028-181209_spike-qkon-lr1e-1",
+  "name": "spike-qkon-lr1e-1",
+  "note": "tech spike smoke test (not an experiment); divergence-detection check",
+  "status": "completed",
+  "started": "2026-10-28T18:12:09+00:00",
+  "command": "/Users/nik/ISE/9.CS6513/thesis/06-experiments/.venv/bin/python /Users/nik/ISE/9.CS6513/thesis/06-experiments/nanogpt/train.py configs/spike.py --out_dir=/Users/nik/ISE/9.CS6513/thesis/06-experiments/runs/20261028-181209_spike-qkon-lr1e-1 --data_root=/Users/nik/ISE/9.CS6513/thesis/06-experiments/data --qk_norm=True --learning_rate=1e-1 --min_lr=1e-2",
+  "config_file": "configs/spike.py",
+  "overrides": [
+    "qk_norm=True",
+    "learning_rate=1e-1",
+    "min_lr=1e-2"
+  ],
+  "config_requested": {
+    "dataset": "tinystories_spike",
+    "device": "mps",
+    "dtype": "float32",
+    "compile": false,
+    "n_layer": 4,
+    "n_head": 4,
+    "n_embd": 256,
+    "block_size": 256,
+    "bias": false,
+    "dropout": 0.0,
+    "qk_norm": true,
+    "batch_size": 32,
+    "gradient_accumulation_steps": 1,
+    "learning_rate": 0.1,
+    "max_iters": 300,
+    "lr_decay_iters": 300,
+    "min_lr": 0.01,
+    "warmup_iters": 30,
+    "weight_decay": 0.1,
+    "grad_clip": 1.0,
+    "eval_interval": 100,
+    "eval_iters": 20,
+    "log_interval": 10,
+    "always_save_checkpoint": false,
+    "seed": 1337
+  },
+  "git": {
+    "commit": "55ba128dcca8310f85e99e172fe920a8c53a3a79",
+    "branch": "main",
+    "dirty": false
+  },
+  "nanogpt_upstream_commit": "3adf61e154c3fe3fca428ad6bc3818b27a3b8291",
+  "data": {
+    "dataset": "TinyStories V2 (GPT-4)",
+    "source": "https://huggingface.co/datasets/roneneldan/TinyStories/resolve/main/",
+    "tokenizer": "tiktoken gpt2",
+    "files": {
+      "TinyStoriesV2-GPT4-valid.txt": "6874bae9a4c1a4e7edcf0e53b86c17817e9cf881fc75ff2368da457b80c0585d"
+    },
+    "split": "valid file split 90/10 by story order: 24867 train / 2763 val stories",
+    "tokens": {
+      "train": 4920360,
+      "val": 548664
+    },
+    "train_bin_bytes": 9840720,
+    "val_bin_bytes": 1097328
+  },
+  "environment": {
+    "host": "Cormacs-MacBook-Pro.local",
+    "platform": "macOS-15.7.7-arm64-arm-64bit",
+    "machine": "arm64",
+    "cpu": "Apple M4 Pro",
+    "python": "3.12.14",
+    "torch": "2.15.0",
+    "mps_available": true,
+    "cuda_available": false
+  },
+  "finished": "2026-10-28T18:13:55+00:00",
+  "wall_seconds": 105.7,
+  "summary": {
+    "params_millions_nanogpt_count": 16.03,
+    "first_train_loss": 10.9141,
+    "last_train_loss": 3.8235,
+    "last_eval": {
+      "iter": 300,
+      "train_loss": 3.8471,
+      "val_loss": 3.9026
+    },
+    "best_val_loss": 3.9026,
+    "diverged_at_iter": null,
+    "returncode": 0
+  }
+}

06-experiments/runs/20261028-181408_test-divergence-stop/config.json

@@ -0,0 +1,38 @@
+{
+  "out_dir": "/Users/nik/ISE/9.CS6513/thesis/06-experiments/runs/20261028-181408_test-divergence-stop",
+  "eval_interval": 1000,
+  "log_interval": 10,
+  "eval_iters": 20,
+  "eval_only": false,
+  "always_save_checkpoint": false,
+  "init_from": "scratch",
+  "wandb_log": false,
+  "wandb_project": "owt",
+  "wandb_run_name": "gpt2",
+  "dataset": "tinystories_spike",
+  "gradient_accumulation_steps": 1,
+  "batch_size": 32,
+  "block_size": 256,
+  "n_layer": 4,
+  "n_head": 4,
+  "n_embd": 256,
+  "dropout": 0.0,
+  "bias": false,
+  "qk_norm": false,
+  "seed": 1337,
+  "data_root": "/Users/nik/ISE/9.CS6513/thesis/06-experiments/data",
+  "learning_rate": 0.1,
+  "max_iters": 60,
+  "weight_decay": 0.1,
+  "beta1": 0.9,
+  "beta2": 0.95,
+  "grad_clip": 1.0,
+  "decay_lr": true,
+  "warmup_iters": 5,
+  "lr_decay_iters": 60,
+  "min_lr": 0.0001,
+  "backend": "nccl",
+  "device": "mps",
+  "dtype": "float32",
+  "compile": false
+}

06-experiments/runs/20261028-181408_test-divergence-stop/metrics.jsonl

@@ -0,0 +1,2 @@
+{"type": "eval", "iter": 0, "train_loss": 10.9141, "val_loss": 10.9167, "t": 5.48}
+{"type": "train", "iter": 0, "loss": 10.9193, "ms": 4405.09, "t": 5.8}

06-experiments/runs/20261028-181408_test-divergence-stop/run.json

@@ -0,0 +1,89 @@
+{
+  "run_id": "20261028-181408_test-divergence-stop",
+  "name": "test-divergence-stop",
+  "note": "plumbing test of the divergence stop (threshold set artificially low); not an experiment",
+  "status": "diverged",
+  "started": "2026-10-28T18:14:08+00:00",
+  "command": "/Users/nik/ISE/9.CS6513/thesis/06-experiments/.venv/bin/python /Users/nik/ISE/9.CS6513/thesis/06-experiments/nanogpt/train.py configs/spike.py --out_dir=/Users/nik/ISE/9.CS6513/thesis/06-experiments/runs/20261028-181408_test-divergence-stop --data_root=/Users/nik/ISE/9.CS6513/thesis/06-experiments/data --learning_rate=1e-1 --max_iters=60 --lr_decay_iters=60 --warmup_iters=5 --eval_interval=1000",
+  "config_file": "configs/spike.py",
+  "overrides": [
+    "learning_rate=1e-1",
+    "max_iters=60",
+    "lr_decay_iters=60",
+    "warmup_iters=5",
+    "eval_interval=1000"
+  ],
+  "config_requested": {
+    "dataset": "tinystories_spike",
+    "device": "mps",
+    "dtype": "float32",
+    "compile": false,
+    "n_layer": 4,
+    "n_head": 4,
+    "n_embd": 256,
+    "block_size": 256,
+    "bias": false,
+    "dropout": 0.0,
+    "qk_norm": false,
+    "batch_size": 32,
+    "gradient_accumulation_steps": 1,
+    "learning_rate": 0.1,
+    "max_iters": 60,
+    "lr_decay_iters": 60,
+    "min_lr": 0.0001,
+    "warmup_iters": 5,
+    "weight_decay": 0.1,
+    "grad_clip": 1.0,
+    "eval_interval": 1000,
+    "eval_iters": 20,
+    "log_interval": 10,
+    "always_save_checkpoint": false,
+    "seed": 1337
+  },
+  "git": {
+    "commit": "55ba128dcca8310f85e99e172fe920a8c53a3a79",
+    "branch": "main",
+    "dirty": false
+  },
+  "nanogpt_upstream_commit": "3adf61e154c3fe3fca428ad6bc3818b27a3b8291",
+  "data": {
+    "dataset": "TinyStories V2 (GPT-4)",
+    "source": "https://huggingface.co/datasets/roneneldan/TinyStories/resolve/main/",
+    "tokenizer": "tiktoken gpt2",
+    "files": {
+      "TinyStoriesV2-GPT4-valid.txt": "6874bae9a4c1a4e7edcf0e53b86c17817e9cf881fc75ff2368da457b80c0585d"
+    },
+    "split": "valid file split 90/10 by story order: 24867 train / 2763 val stories",
+    "tokens": {
+      "train": 4920360,
+      "val": 548664
+    },
+    "train_bin_bytes": 9840720,
+    "val_bin_bytes": 1097328
+  },
+  "environment": {
+    "host": "Cormacs-MacBook-Pro.local",
+    "platform": "macOS-15.7.7-arm64-arm-64bit",
+    "machine": "arm64",
+    "cpu": "Apple M4 Pro",
+    "python": "3.12.14",
+    "torch": "2.15.0",
+    "mps_available": true,
+    "cuda_available": false
+  },
+  "finished": "2026-10-28T18:14:15+00:00",
+  "wall_seconds": 5.8,
+  "summary": {
+    "params_millions_nanogpt_count": 16.03,
+    "first_train_loss": 10.9193,
+    "last_train_loss": 10.9193,
+    "last_eval": {
+      "iter": 0,
+      "train_loss": 10.9141,
+      "val_loss": 10.9167
+    },
+    "best_val_loss": 10.9167,
+    "diverged_at_iter": 0,
+    "returncode": -15
+  }
+}

06-experiments/runs/index.csv

@@ -0,0 +1,6 @@
+run_id,status,qk_norm,learning_rate,n_layer,n_embd,seed,max_iters,params_M,best_val_loss,diverged_at,wall_s,git_commit,dirty,note
+20261028-180656_spike-qkoff-lr1e-3,completed,False,0.001,4,256,1337,300,16.03,3.459,,102.9,55ba128dcc,False,tech spike smoke test (not an experiment)
+20261028-180840_spike-qkon-lr1e-3,completed,True,0.001,4,256,1337,300,16.03,3.4304,,104.8,55ba128dcc,False,tech spike smoke test (not an experiment)
+20261028-181025_spike-qkoff-lr1e-1,completed,False,0.1,4,256,1337,300,16.03,5.2948,,103.4,55ba128dcc,False,tech spike smoke test (not an experiment); divergence-detection check
+20261028-181209_spike-qkon-lr1e-1,completed,True,0.1,4,256,1337,300,16.03,3.9026,,105.7,55ba128dcc,False,tech spike smoke test (not an experiment); divergence-detection check
+20261028-181408_test-divergence-stop,diverged,False,0.1,4,256,1337,60,16.03,10.9167,0,5.8,55ba128dcc,False,plumbing test of the divergence stop (threshold set artificially low); not an experiment