Cormac Shannon
95b5857189
Six seeds snowballed (Semantic Scholar) plus logged keyword searches (OpenAlex, arXiv): 2,127 unique records, 480 title/abstract screened. AI first-pass suggestions are in their own column; decisions are pending. Claude-Session: https://claude.ai/code/session_01S8KN7NuUCm8nXLmy6eDuVv
199 lines
12 KiB
Markdown
199 lines
12 KiB
Markdown
# Literature Review Protocol (pared-down SLR)
|
||
|
||
*Version 0.1 — drafted 28/10/2026 with AI assistance (see section 9). Status: DRAFT for
|
||
Cormac to review and own. `[Your call: …]` marks decisions that are yours to make.*
|
||
|
||
Follows a pared-down version of Kitchenham-style SLR guidance, as summarised in
|
||
Xiao & Watson (2017, *J. Planning Education and Research*, doi:10.1177/0739456X17723971 —
|
||
in `05-course-materials/week2-literature-review/`), with snowballing per
|
||
Wohlin (2014, EASE, doi:10.1145/2601248.2601268).
|
||
|
||
---
|
||
|
||
## 1. Review questions
|
||
|
||
Supervisor advice: ask **how**, not **if** — "how" questions give a scope that can be
|
||
answered with a set of findings, rather than a yes/no.
|
||
|
||
| ID | Review question (draft) | Feeds |
|
||
|---|---|---|
|
||
| RQ-L1 | **How** has QK-norm (and related attention-logit control) been evaluated with respect to training stability and learning-rate sensitivity, and at what model scales? | Gap: is <100M covered? |
|
||
| RQ-L2 | **How** do reported benefits of QK-norm depend on learning rate, depth and model size? | H1, H2, H3 |
|
||
| RQ-L3 | **How** do existing studies isolate QK-norm from confounds (μP, z-loss, soft-capping, optimiser, precision)? | "isolated" claim |
|
||
| RQ-L4 | **How** do authors explain *why* QK-norm stabilises training (logit growth, entropy collapse, sinks, kurtosis)? | metrics to log |
|
||
|
||
**Thesis RQ (current, from notebook 21/08/2026):** "At <100M parameters, *does* isolated QK-norm
|
||
improve training convergence and raise the maximum stable learning rate — and at what learning
|
||
rate does it become necessary?"
|
||
|
||
`[Your call: possible "how" rewrite — "How does isolated QK-norm change the maximum stable
|
||
learning rate and LR-sensitivity of <100M-parameter language models, and how does the learning
|
||
rate at which it becomes necessary vary with depth?" Revisit after the extraction table is
|
||
filled (supervisor: "review if you want to update/change any of your questions").]`
|
||
|
||
## 2. Sources
|
||
|
||
| Source | Used for | Notes |
|
||
|---|---|---|
|
||
| Semantic Scholar Graph API | snowballing (references + citations) | best citation graph for ML preprints; unauthenticated access is heavily rate-limited (a free key helps) |
|
||
| OpenAlex API | keyword search | open, free; **no reference lists for arXiv preprints**, so not used for snowballing |
|
||
| arXiv API | keyword search | where nearly all of this literature appears first |
|
||
| Google Scholar | manual check only | no API; use to spot-check that nothing big is missing `[Your call: run the S1 string by hand and log the top 50]` |
|
||
|
||
All automated searches are logged in `search/search-log.csv` (date, source, exact query,
|
||
filters, hit count, results file). Raw API responses are cached in `search/.cache/`
|
||
(git-ignored) so every search can be re-run and gives the same results.
|
||
|
||
## 3. Seed set
|
||
|
||
Chosen from the CS6513 lit review (`qknorm-research-findings.md`) as the papers the thesis
|
||
depends on most:
|
||
|
||
| Key | Paper | Why it is a seed |
|
||
|---|---|---|
|
||
| henry2020 | Henry et al. 2020, *Query-Key Normalization for Transformers* (Findings of EMNLP) | origin of the technique |
|
||
| vit22b | Dehghani et al. 2023, *Scaling Vision Transformers to 22B Parameters* | popularised QK-norm for stability |
|
||
| zhai2023 | Zhai et al. 2023, *Preventing Attention Entropy Collapse* (ICML) | mechanism |
|
||
| wortsman2024 | Wortsman et al. 2023, *Small-scale proxies for large-scale Transformer training instabilities* | closest method template (LR-sensitivity) |
|
||
| imu1 | Grigorev 2026, *IMU-1* | closest small-scale evidence (70M) |
|
||
| stability2024 | Rybakov, Chrzanowski, Dykas, Xue et al. 2024, *Methods of improving LLM training stability* (arXiv:2410.16682) | QK-norm → 1.5× LR before divergence |
|
||
|
||
Authors of all six seeds checked against arXiv (28/10/2026).
|
||
|
||
## 4. Search strings
|
||
|
||
Concepts: **(A) technique** × **(B) outcome** × **(C) scale**. Phrases are quoted; `|` = OR.
|
||
|
||
| ID | Source | String |
|
||
|---|---|---|
|
||
| S1 | OpenAlex (title+abstract) | `"qk-norm" \| "qk norm" \| "qk-layernorm" \| "qk layernorm" \| "query-key normalization" \| "query key normalization" \| "qk normalization"` |
|
||
| S2 | OpenAlex | (`"attention logit" \| "attention logits" \| "attention entropy collapse" \| "logit growth"`) AND (`"training instability" \| "training stability" \| "loss spike" \| "divergence" \| "learning rate"`) |
|
||
| S3 | OpenAlex | (`"small language model(s)" \| "small-scale proxies" \| "small transformer"`) AND (`"learning rate sensitivity" \| "training instability" \| "training stability"`) AND `"normalization"` |
|
||
| A1 | arXiv | `abs:"qk norm" OR abs:"qk-norm" OR abs:"query-key normalization" OR abs:"qk normalization" OR abs:"qk-layernorm" OR abs:"qk layernorm"` |
|
||
| A2 | arXiv | `(abs:"attention logit" OR abs:"attention logits" OR abs:"entropy collapse") AND (abs:"training instability" OR abs:"learning rate" OR abs:"loss spike")` |
|
||
| L1 | OpenAlex, ≥2023 | LLM terms AND (`"systematic literature review" \| "systematic review" \| "literature search" \| "literature screening"`) AND (`"search" \| "screening"`) — *for section 9* |
|
||
| L2 | OpenAlex, ≥2023 | LLM terms AND (`"systematic literature review" \| "literature review" \| "snowballing"`) AND `"software engineering"` — *for section 9* |
|
||
|
||
S3 returned only 1 hit, so the "small scale" concept is covered mainly by snowballing and by
|
||
the extraction table's model-size column rather than by the search string.
|
||
|
||
## 5. Inclusion / exclusion criteria
|
||
|
||
**Include** if all of:
|
||
- **I1** published 2020-01-01 or later (QK-norm was introduced in Oct 2020; nothing earlier is
|
||
about it). Pre-2020 *background* (e.g. Vaswani et al. 2017) may be cited, but is not a search result.
|
||
- **I2** about transformer models, and addresses QK-norm, attention-logit control, or the
|
||
training stability / LR-sensitivity of transformers.
|
||
- **I3** English, full text available.
|
||
|
||
**Exclude** codes used in `search/screening.csv`:
|
||
|
||
| Code | Meaning |
|
||
|---|---|
|
||
| E1 | applies a transformer to another domain (vision, speech, recommendation, medicine…); QK-norm is just one part of the architecture and its effect isn't studied |
|
||
| E2 | inference / systems / hardware only (kernels, KV cache, tensor parallelism) |
|
||
| E3 | "entropy collapse" in the RL / policy sense, not attention |
|
||
| E4 | duplicate, earlier title of an included paper, or a citation fragment rather than a paper |
|
||
| E5 | off-topic on title/abstract |
|
||
| E6 | (LLM-search set) uses LLMs as the research *subject*, not as a review tool |
|
||
|
||
**Include** codes (why it's in):
|
||
|
||
| Code | Meaning |
|
||
|---|---|
|
||
| SEED | seed paper |
|
||
| IC | evaluates QK-norm's effect on stability / LR / loss |
|
||
| IA | alternative or complementary stability method, usually compared against QK-norm |
|
||
| IM | mechanism: why logits grow, entropy collapse, attention sinks, massive activations |
|
||
| IS | small-scale controlled ablation study (method precedent) |
|
||
| BG | background / confound (μP, LR transfer, Pre-LN) |
|
||
| LLM | evidence on using LLMs to support literature reviews (section 9) |
|
||
| `*` | **close to the thesis RQ** — needs "how my study differs" in the extraction table |
|
||
|
||
## 6. Snowballing (Wohlin 2014)
|
||
|
||
1. **Backward**: every reference of each seed ("what did these guys cite?").
|
||
2. **Forward**: every paper citing each seed.
|
||
3. Apply the criteria (I1 automatically; I2 first by keyword filter, then by title/abstract).
|
||
4. Each newly *included* paper becomes a seed for the next iteration. Stop when an iteration
|
||
adds nothing new.
|
||
|
||
**Status:** iteration 1 done for all six seeds (28/10/2026). Iteration 2 not started —
|
||
`[Your call]` which of the iteration-1 includes to snowball from (suggest the `*` ones).
|
||
|
||
## 7. Screening procedure and flow (iteration 1)
|
||
|
||
1. Automatic: de-duplicate by arXiv ID / DOI / title (`tools/merge.py`).
|
||
2. Automatic: drop pre-2020 (I1) and records failing a keyword filter for I2.
|
||
3. **AI first pass** on title/abstract (`tools/ai_first_pass.py`) → `ai_proposal` column.
|
||
4. **Cormac decides** → `decision` column. Disagreements with the AI pass are worth noting —
|
||
they're evidence for section 9.
|
||
5. Full-text read of includes → extraction table.
|
||
|
||
| Stage | QK-norm review | LLM-for-review set |
|
||
|---|---|---|
|
||
| Records retrieved (all searches + snowballing) | 2,127 unique, after de-duplication | (included in the 2,127) |
|
||
| Excluded automatically: pre-2020 | 129 | — |
|
||
| Excluded automatically: off-topic keyword filter | 1,518 (both sets) | |
|
||
| Title/abstract screened | 237 | 243 |
|
||
| AI proposal: include / maybe / exclude | 33 / 25 / 179 | 6 / 6 / 231 |
|
||
| **Your decisions** | `[pending]` | `[pending]` |
|
||
| Full text read | `[pending]` | `[pending]` |
|
||
|
||
## 8. Data extraction
|
||
|
||
Template: `extraction-table.csv`. One row per included paper. Columns:
|
||
key, citation, year, venue/status, model sizes, data, **question asked**, **main claim**,
|
||
method (what's varied / held fixed), QK-norm isolated? (y/n/what bundled), μP used?,
|
||
LR sweep (points, range), depth varied?, metrics, close to my RQ (`*`),
|
||
**how my study differs (≤2 sentences)**, notes.
|
||
|
||
Synthesis questions to answer from the table:
|
||
- what questions are people asking across the literature?
|
||
- which papers are close to the thesis RQ, and for each: two sentences on why the thesis is different
|
||
(e.g. smaller scale, μP off, single variable, laptop cost, dense sweep vs 3 points)?
|
||
|
||
## 9. Use of LLMs in this review (and the paper that backs it)
|
||
|
||
Supervisor: "if you want to use LLMs for paper search, find a paper that backs up the tactic."
|
||
|
||
**What an LLM (Claude Code) was used for:** writing the search scripts; drafting this
|
||
protocol and the search strings; a first-pass title/abstract screen (suggestions only, kept in a
|
||
separate column); checking citation metadata against arXiv/Crossref. **Not** used for: final
|
||
include decisions, the extraction of claims, or writing thesis text.
|
||
|
||
**Supporting literature** (all metadata checked against Crossref or arXiv, 28/10/2026):
|
||
|
||
- **Kitchenham, Pizard, Madeyski, de Souza Santos, Shepperd (2026)**, *Preliminary Guidelines for
|
||
Using and Evaluating GenAI Tools to Support Systematic Literature Reviews*, arXiv:2607.24991.
|
||
The GUEST recommendations. Conclude GenAI "requires human oversight and is not currently capable
|
||
of unsupervised systematic studies" but offers "cost-effective assistance for some repetitive
|
||
tasks". **Main citation for the approach taken here** (human makes every decision; AI suggests).
|
||
- Felizardo et al. (2024), *ChatGPT application in SLRs in Software Engineering: an evaluation of its
|
||
accuracy to support the selection activity*, ESEM 2024, doi:10.1145/3674805.3686666.
|
||
- Felizardo et al. (2025), *On the Difficulties of Conducting and Replicating SLR Studies Using LLMs
|
||
in Software Engineering*, WSESE 2025, doi:10.1109/WSESE66602.2025.00010 — why the search log
|
||
and fixed scripts matter.
|
||
- Wang, Scells, Koopman, Zuccon (2023), *Can ChatGPT Write a Good Boolean Query for Systematic Review
|
||
Literature Search?*, SIGIR 2023, doi:10.1145/3539618.3591703 — supports LLM help with search strings.
|
||
- Khraisha et al. (2024), *Can large language models replace humans in systematic reviews?*,
|
||
Research Synthesis Methods, doi:10.1002/jrsm.1715 — caution: GPT-4 screening performance drops
|
||
sharply once chance agreement is accounted for.
|
||
- Lieberum et al. (2025), *Large language models for conducting systematic reviews: on the rise, but
|
||
not yet ready for use — a scoping review*, J. Clinical Epidemiology, doi:10.1016/j.jclinepi.2025.111746.
|
||
|
||
`[Your note: in your own words, why this level of LLM use is defensible given the above.]`
|
||
|
||
## 10. Deviations log
|
||
|
||
| Date | Planned | Actual | Why |
|
||
|---|---|---|---|
|
||
| 28/10/2026 | Semantic Scholar for all searching | Keyword searches via OpenAlex + arXiv; S2 for snowballing only | S2 unauthenticated API returned HTTP 429 on search; OpenAlex lacks arXiv reference lists |
|
||
|
||
## 11. Defending the review (for the viva / interview)
|
||
|
||
"This is how I found these papers": six seeds → one round of forward/backward snowballing on
|
||
Semantic Scholar → keyword searches on arXiv and OpenAlex with logged strings → automatic date
|
||
and topic filters → title/abstract screening with written reasons → full-text extraction.
|
||
Every step is a script plus a log, so it can be re-run.
|