Add literature review protocol, searches and first snowball round

Six seeds snowballed (Semantic Scholar) plus logged keyword searches
(OpenAlex, arXiv): 2,127 unique records, 480 title/abstract screened.
AI first-pass suggestions are in their own column; decisions are pending.

Claude-Session: https://claude.ai/code/session_01S8KN7NuUCm8nXLmy6eDuVv

02-literature/protocol.md

@@ -0,0 +1,198 @@
+# Literature Review Protocol (pared-down SLR)
+
+*Version 0.1 — drafted 28/10/2026 with AI assistance (see section 9). Status: DRAFT for
+Cormac to review and own. `[Your call: …]` marks decisions that are yours to make.*
+
+Follows a pared-down version of Kitchenham-style SLR guidance, as summarised in
+Xiao & Watson (2017, *J. Planning Education and Research*, doi:10.1177/0739456X17723971 —
+in `05-course-materials/week2-literature-review/`), with snowballing per
+Wohlin (2014, EASE, doi:10.1145/2601248.2601268).
+
+---
+
+## 1. Review questions
+
+Supervisor advice: ask **how**, not **if** — "how" questions give a scope that can be
+answered with a set of findings, rather than a yes/no.
+
+| ID | Review question (draft) | Feeds |
+|---|---|---|
+| RQ-L1 | **How** has QK-norm (and related attention-logit control) been evaluated with respect to training stability and learning-rate sensitivity, and at what model scales? | Gap: is <100M covered? |
+| RQ-L2 | **How** do reported benefits of QK-norm depend on learning rate, depth and model size? | H1, H2, H3 |
+| RQ-L3 | **How** do existing studies isolate QK-norm from confounds (μP, z-loss, soft-capping, optimiser, precision)? | "isolated" claim |
+| RQ-L4 | **How** do authors explain *why* QK-norm stabilises training (logit growth, entropy collapse, sinks, kurtosis)? | metrics to log |
+
+**Thesis RQ (current, from notebook 21/08/2026):** "At <100M parameters, *does* isolated QK-norm
+improve training convergence and raise the maximum stable learning rate — and at what learning
+rate does it become necessary?"
+
+`[Your call: possible "how" rewrite — "How does isolated QK-norm change the maximum stable
+learning rate and LR-sensitivity of <100M-parameter language models, and how does the learning
+rate at which it becomes necessary vary with depth?" Revisit after the extraction table is
+filled (supervisor: "review if you want to update/change any of your questions").]`
+
+## 2. Sources
+
+| Source | Used for | Notes |
+|---|---|---|
+| Semantic Scholar Graph API | snowballing (references + citations) | best citation graph for ML preprints; unauthenticated access is heavily rate-limited (a free key helps) |
+| OpenAlex API | keyword search | open, free; **no reference lists for arXiv preprints**, so not used for snowballing |
+| arXiv API | keyword search | where nearly all of this literature appears first |
+| Google Scholar | manual check only | no API; use to spot-check that nothing big is missing `[Your call: run the S1 string by hand and log the top 50]` |
+
+All automated searches are logged in `search/search-log.csv` (date, source, exact query,
+filters, hit count, results file). Raw API responses are cached in `search/.cache/`
+(git-ignored) so every search can be re-run and gives the same results.
+
+## 3. Seed set
+
+Chosen from the CS6513 lit review (`qknorm-research-findings.md`) as the papers the thesis
+depends on most:
+
+| Key | Paper | Why it is a seed |
+|---|---|---|
+| henry2020 | Henry et al. 2020, *Query-Key Normalization for Transformers* (Findings of EMNLP) | origin of the technique |
+| vit22b | Dehghani et al. 2023, *Scaling Vision Transformers to 22B Parameters* | popularised QK-norm for stability |
+| zhai2023 | Zhai et al. 2023, *Preventing Attention Entropy Collapse* (ICML) | mechanism |
+| wortsman2024 | Wortsman et al. 2023, *Small-scale proxies for large-scale Transformer training instabilities* | closest method template (LR-sensitivity) |
+| imu1 | Grigorev 2026, *IMU-1* | closest small-scale evidence (70M) |
+| stability2024 | Rybakov, Chrzanowski, Dykas, Xue et al. 2024, *Methods of improving LLM training stability* (arXiv:2410.16682) | QK-norm → 1.5× LR before divergence |
+
+Authors of all six seeds checked against arXiv (28/10/2026).
+
+## 4. Search strings
+
+Concepts: **(A) technique** × **(B) outcome** × **(C) scale**. Phrases are quoted; `|` = OR.
+
+| ID | Source | String |
+|---|---|---|
+| S1 | OpenAlex (title+abstract) | `"qk-norm" \| "qk norm" \| "qk-layernorm" \| "qk layernorm" \| "query-key normalization" \| "query key normalization" \| "qk normalization"` |
+| S2 | OpenAlex | (`"attention logit" \| "attention logits" \| "attention entropy collapse" \| "logit growth"`) AND (`"training instability" \| "training stability" \| "loss spike" \| "divergence" \| "learning rate"`) |
+| S3 | OpenAlex | (`"small language model(s)" \| "small-scale proxies" \| "small transformer"`) AND (`"learning rate sensitivity" \| "training instability" \| "training stability"`) AND `"normalization"` |
+| A1 | arXiv | `abs:"qk norm" OR abs:"qk-norm" OR abs:"query-key normalization" OR abs:"qk normalization" OR abs:"qk-layernorm" OR abs:"qk layernorm"` |
+| A2 | arXiv | `(abs:"attention logit" OR abs:"attention logits" OR abs:"entropy collapse") AND (abs:"training instability" OR abs:"learning rate" OR abs:"loss spike")` |
+| L1 | OpenAlex, ≥2023 | LLM terms AND (`"systematic literature review" \| "systematic review" \| "literature search" \| "literature screening"`) AND (`"search" \| "screening"`) — *for section 9* |
+| L2 | OpenAlex, ≥2023 | LLM terms AND (`"systematic literature review" \| "literature review" \| "snowballing"`) AND `"software engineering"` — *for section 9* |
+
+S3 returned only 1 hit, so the "small scale" concept is covered mainly by snowballing and by
+the extraction table's model-size column rather than by the search string.
+
+## 5. Inclusion / exclusion criteria
+
+**Include** if all of:
+- **I1** published 2020-01-01 or later (QK-norm was introduced in Oct 2020; nothing earlier is
+  about it). Pre-2020 *background* (e.g. Vaswani et al. 2017) may be cited, but is not a search result.
+- **I2** about transformer models, and addresses QK-norm, attention-logit control, or the
+  training stability / LR-sensitivity of transformers.
+- **I3** English, full text available.
+
+**Exclude** codes used in `search/screening.csv`:
+
+| Code | Meaning |
+|---|---|
+| E1 | applies a transformer to another domain (vision, speech, recommendation, medicine…); QK-norm is just one part of the architecture and its effect isn't studied |
+| E2 | inference / systems / hardware only (kernels, KV cache, tensor parallelism) |
+| E3 | "entropy collapse" in the RL / policy sense, not attention |
+| E4 | duplicate, earlier title of an included paper, or a citation fragment rather than a paper |
+| E5 | off-topic on title/abstract |
+| E6 | (LLM-search set) uses LLMs as the research *subject*, not as a review tool |
+
+**Include** codes (why it's in):
+
+| Code | Meaning |
+|---|---|
+| SEED | seed paper |
+| IC | evaluates QK-norm's effect on stability / LR / loss |
+| IA | alternative or complementary stability method, usually compared against QK-norm |
+| IM | mechanism: why logits grow, entropy collapse, attention sinks, massive activations |
+| IS | small-scale controlled ablation study (method precedent) |
+| BG | background / confound (μP, LR transfer, Pre-LN) |
+| LLM | evidence on using LLMs to support literature reviews (section 9) |
+| `*` | **close to the thesis RQ** — needs "how my study differs" in the extraction table |
+
+## 6. Snowballing (Wohlin 2014)
+
+1. **Backward**: every reference of each seed ("what did these guys cite?").
+2. **Forward**: every paper citing each seed.
+3. Apply the criteria (I1 automatically; I2 first by keyword filter, then by title/abstract).
+4. Each newly *included* paper becomes a seed for the next iteration. Stop when an iteration
+   adds nothing new.
+
+**Status:** iteration 1 done for all six seeds (28/10/2026). Iteration 2 not started —
+`[Your call]` which of the iteration-1 includes to snowball from (suggest the `*` ones).
+
+## 7. Screening procedure and flow (iteration 1)
+
+1. Automatic: de-duplicate by arXiv ID / DOI / title (`tools/merge.py`).
+2. Automatic: drop pre-2020 (I1) and records failing a keyword filter for I2.
+3. **AI first pass** on title/abstract (`tools/ai_first_pass.py`) → `ai_proposal` column.
+4. **Cormac decides** → `decision` column. Disagreements with the AI pass are worth noting —
+   they're evidence for section 9.
+5. Full-text read of includes → extraction table.
+
+| Stage | QK-norm review | LLM-for-review set |
+|---|---|---|
+| Records retrieved (all searches + snowballing) | 2,127 unique, after de-duplication | (included in the 2,127) |
+| Excluded automatically: pre-2020 | 129 | — |
+| Excluded automatically: off-topic keyword filter | 1,518 (both sets) | |
+| Title/abstract screened | 237 | 243 |
+| AI proposal: include / maybe / exclude | 33 / 25 / 179 | 6 / 6 / 231 |
+| **Your decisions** | `[pending]` | `[pending]` |
+| Full text read | `[pending]` | `[pending]` |
+
+## 8. Data extraction
+
+Template: `extraction-table.csv`. One row per included paper. Columns:
+key, citation, year, venue/status, model sizes, data, **question asked**, **main claim**,
+method (what's varied / held fixed), QK-norm isolated? (y/n/what bundled), μP used?,
+LR sweep (points, range), depth varied?, metrics, close to my RQ (`*`),
+**how my study differs (≤2 sentences)**, notes.
+
+Synthesis questions to answer from the table:
+- what questions are people asking across the literature?
+- which papers are close to the thesis RQ, and for each: two sentences on why the thesis is different
+  (e.g. smaller scale, μP off, single variable, laptop cost, dense sweep vs 3 points)?
+
+## 9. Use of LLMs in this review (and the paper that backs it)
+
+Supervisor: "if you want to use LLMs for paper search, find a paper that backs up the tactic."
+
+**What an LLM (Claude Code) was used for:** writing the search scripts; drafting this
+protocol and the search strings; a first-pass title/abstract screen (suggestions only, kept in a
+separate column); checking citation metadata against arXiv/Crossref. **Not** used for: final
+include decisions, the extraction of claims, or writing thesis text.
+
+**Supporting literature** (all metadata checked against Crossref or arXiv, 28/10/2026):
+
+- **Kitchenham, Pizard, Madeyski, de Souza Santos, Shepperd (2026)**, *Preliminary Guidelines for
+  Using and Evaluating GenAI Tools to Support Systematic Literature Reviews*, arXiv:2607.24991.
+  The GUEST recommendations. Conclude GenAI "requires human oversight and is not currently capable
+  of unsupervised systematic studies" but offers "cost-effective assistance for some repetitive
+  tasks". **Main citation for the approach taken here** (human makes every decision; AI suggests).
+- Felizardo et al. (2024), *ChatGPT application in SLRs in Software Engineering: an evaluation of its
+  accuracy to support the selection activity*, ESEM 2024, doi:10.1145/3674805.3686666.
+- Felizardo et al. (2025), *On the Difficulties of Conducting and Replicating SLR Studies Using LLMs
+  in Software Engineering*, WSESE 2025, doi:10.1109/WSESE66602.2025.00010 — why the search log
+  and fixed scripts matter.
+- Wang, Scells, Koopman, Zuccon (2023), *Can ChatGPT Write a Good Boolean Query for Systematic Review
+  Literature Search?*, SIGIR 2023, doi:10.1145/3539618.3591703 — supports LLM help with search strings.
+- Khraisha et al. (2024), *Can large language models replace humans in systematic reviews?*,
+  Research Synthesis Methods, doi:10.1002/jrsm.1715 — caution: GPT-4 screening performance drops
+  sharply once chance agreement is accounted for.
+- Lieberum et al. (2025), *Large language models for conducting systematic reviews: on the rise, but
+  not yet ready for use — a scoping review*, J. Clinical Epidemiology, doi:10.1016/j.jclinepi.2025.111746.
+
+`[Your note: in your own words, why this level of LLM use is defensible given the above.]`
+
+## 10. Deviations log
+
+| Date | Planned | Actual | Why |
+|---|---|---|---|
+| 28/10/2026 | Semantic Scholar for all searching | Keyword searches via OpenAlex + arXiv; S2 for snowballing only | S2 unauthenticated API returned HTTP 429 on search; OpenAlex lacks arXiv reference lists |
+
+## 11. Defending the review (for the viva / interview)
+
+"This is how I found these papers": six seeds → one round of forward/backward snowballing on
+Semantic Scholar → keyword searches on arXiv and OpenAlex with logged strings → automatic date
+and topic filters → title/abstract screening with written reasons → full-text extraction.
+Every step is a script plus a log, so it can be re-run.