Tech spike: four smoke-test runs and a divergence-stop test
Pipeline works on MPS (~105 s per 300-step run), results are reproducible, and the divergence stop works. Not experimental evidence. Claude-Session: https://claude.ai/code/session_01S8KN7NuUCm8nXLmy6eDuVv
This commit is contained in:
1 parent
55ba128dcc
commit
84b04d12e9
17 files changed
+792
-1
No files matched your search
@@ -43,7 +43,25 @@ iteration where it diverged).
|
||||
|
||||
## Tech-spike results (28/10/2026)
|
||||
|
||||
See `runs/index.csv`. Pipeline check only: tiny model, 300 steps, small data subset.
|
||||
See `runs/index.csv`. Pipeline check only: 4 layers × 256 wide, 300 steps, 4.9M-token
|
||||
subset, one seed. **Not evidence for or against any hypothesis.**
|
||||
|
||||
| Run | QK-norm | LR | Best val loss | Time |
|
||||
|---|---|---|---|---|
|
||||
| spike-qkoff-lr1e-3 | off | 1e-3 | 3.459 | 103 s |
|
||||
| spike-qkon-lr1e-3 | on | 1e-3 | 3.430 | 105 s |
|
||||
| spike-qkoff-lr1e-1 | off | 1e-1 | 5.295 | 103 s |
|
||||
| spike-qkon-lr1e-1 | on | 1e-1 | 3.903 | 106 s |
|
||||
| test-divergence-stop | off | 1e-1 | (stopped) | 6 s |
|
||||
|
||||
What the spike showed:
|
||||
- Training runs on MPS at ~0.29 s/step for this size; the whole pipeline works.
|
||||
- **Reproducible:** re-running QK-off at 1e-3 gave the identical val loss (3.4590) as a pre-commit test run.
|
||||
- At LR 0.1, QK-off was clearly worse but **did not diverge**: grad clipping (1.0) kept loss finite.
|
||||
So "diverged" needs a definition beyond NaN for the pre-registration (e.g. final loss vs a threshold).
|
||||
- The divergence stop works (tested with an artificially low threshold; the run was killed and logged).
|
||||
- The QK-on vs off direction at high LR matches H2, but this is one seed, 300 steps: a hint for the
|
||||
pilot, nothing more.
|
||||
|
||||
## Known issues / decisions still open
|
||||
|
||||
|
||||
Reference in new issue
Block a user