Tech spike: four smoke-test runs and a divergence-stop test

Pipeline works on MPS (~105 s per 300-step run), results are reproducible,
and the divergence stop works. Not experimental evidence.

Claude-Session: https://claude.ai/code/session_01S8KN7NuUCm8nXLmy6eDuVv
This commit is contained in:
Cormac Shannon committed 2026-10-28 18:14:26 +00:00
1 parent 55ba128dcc
commit 84b04d12e9
17 files changed
+792 -1

No files matched your search

+19 -1
View File
@@ -43,7 +43,25 @@ iteration where it diverged).
## Tech-spike results (28/10/2026)
See `runs/index.csv`. Pipeline check only: tiny model, 300 steps, small data subset.
See `runs/index.csv`. Pipeline check only: 4 layers × 256 wide, 300 steps, 4.9M-token
subset, one seed. **Not evidence for or against any hypothesis.**
| Run | QK-norm | LR | Best val loss | Time |
|---|---|---|---|---|
| spike-qkoff-lr1e-3 | off | 1e-3 | 3.459 | 103 s |
| spike-qkon-lr1e-3 | on | 1e-3 | 3.430 | 105 s |
| spike-qkoff-lr1e-1 | off | 1e-1 | 5.295 | 103 s |
| spike-qkon-lr1e-1 | on | 1e-1 | 3.903 | 106 s |
| test-divergence-stop | off | 1e-1 | (stopped) | 6 s |
What the spike showed:
- Training runs on MPS at ~0.29 s/step for this size; the whole pipeline works.
- **Reproducible:** re-running QK-off at 1e-3 gave the identical val loss (3.4590) as a pre-commit test run.
- At LR 0.1, QK-off was clearly worse but **did not diverge**: grad clipping (1.0) kept loss finite.
So "diverged" needs a definition beyond NaN for the pre-registration (e.g. final loss vs a threshold).
- The divergence stop works (tested with an artificially low threshold; the run was killed and logged).
- The QK-on vs off direction at high LR matches H2, but this is one seed, 300 steps: a hint for the
pilot, nothing more.
## Known issues / decisions still open