Masters_Thesis/02-literature/protocol.md

Literature Review Protocol (pared-down SLR)

Version 0.1 — drafted 28/10/2026 with AI assistance (see section 9). Status: DRAFT for Cormac to review and own. [Your call: …] marks decisions that are yours to make.

Follows a pared-down version of Kitchenham-style SLR guidance, as summarised in Xiao & Watson (2017, J. Planning Education and Research, doi:10.1177/0739456X17723971 — in 05-course-materials/week2-literature-review/), with snowballing per Wohlin (2014, EASE, doi:10.1145/2601248.2601268).

---

1. Review questions

Supervisor advice: ask how, not if — "how" questions give a scope that can be answered with a set of findings, rather than a yes/no.

ID	Review question (draft)	Feeds
RQ-L1	How has QK-norm (and related attention-logit control) been evaluated with respect to training stability and learning-rate sensitivity, and at what model scales?	Gap: is <100M covered?
RQ-L2	How do reported benefits of QK-norm depend on learning rate, depth and model size?	H1, H2, H3
RQ-L3	How do existing studies isolate QK-norm from confounds (μP, z-loss, soft-capping, optimiser, precision)?	"isolated" claim
RQ-L4	How do authors explain why QK-norm stabilises training (logit growth, entropy collapse, sinks, kurtosis)?	metrics to log

Thesis RQ (current, from notebook 21/08/2026): "At <100M parameters, does isolated QK-norm improve training convergence and raise the maximum stable learning rate — and at what learning rate does it become necessary?"

[Your call: possible "how" rewrite — "How does isolated QK-norm change the maximum stable learning rate and LR-sensitivity of <100M-parameter language models, and how does the learning rate at which it becomes necessary vary with depth?" Revisit after the extraction table is filled (supervisor: "review if you want to update/change any of your questions").]

2. Sources

Source	Used for	Notes
Semantic Scholar Graph API	snowballing (references + citations)	best citation graph for ML preprints; unauthenticated access is heavily rate-limited (a free key helps)
OpenAlex API	keyword search	open, free; no reference lists for arXiv preprints, so not used for snowballing
arXiv API	keyword search	where nearly all of this literature appears first
Google Scholar	manual check only	no API; use to spot-check that nothing big is missing [Your call: run the S1 string by hand and log the top 50]

All automated searches are logged in search/search-log.csv (date, source, exact query, filters, hit count, results file). Raw API responses are cached in search/.cache/ (git-ignored) so every search can be re-run and gives the same results.

3. Seed set

Chosen from the CS6513 lit review (qknorm-research-findings.md) as the papers the thesis depends on most:

Key	Paper	Why it is a seed
henry2020	Henry et al. 2020, Query-Key Normalization for Transformers (Findings of EMNLP)	origin of the technique
vit22b	Dehghani et al. 2023, Scaling Vision Transformers to 22B Parameters	popularised QK-norm for stability
zhai2023	Zhai et al. 2023, Preventing Attention Entropy Collapse (ICML)	mechanism
wortsman2024	Wortsman et al. 2023, Small-scale proxies for large-scale Transformer training instabilities	closest method template (LR-sensitivity)
imu1	Grigorev 2026, IMU-1	closest small-scale evidence (70M)
stability2024	Rybakov, Chrzanowski, Dykas, Xue et al. 2024, Methods of improving LLM training stability (arXiv:2410.16682)	QK-norm → 1.5× LR before divergence

Authors of all six seeds checked against arXiv (28/10/2026).

4. Search strings

Concepts: (A) technique × (B) outcome × (C) scale. Phrases are quoted; | = OR.

ID	Source	String
S1	OpenAlex (title+abstract)	"qk-norm" | "qk norm" | "qk-layernorm" | "qk layernorm" | "query-key normalization" | "query key normalization" | "qk normalization"
S2	OpenAlex	("attention logit" | "attention logits" | "attention entropy collapse" | "logit growth") AND ("training instability" | "training stability" | "loss spike" | "divergence" | "learning rate")
S3	OpenAlex	("small language model(s)" | "small-scale proxies" | "small transformer") AND ("learning rate sensitivity" | "training instability" | "training stability") AND "normalization"
A1	arXiv	abs:"qk norm" OR abs:"qk-norm" OR abs:"query-key normalization" OR abs:"qk normalization" OR abs:"qk-layernorm" OR abs:"qk layernorm"
A2	arXiv	(abs:"attention logit" OR abs:"attention logits" OR abs:"entropy collapse") AND (abs:"training instability" OR abs:"learning rate" OR abs:"loss spike")
L1	OpenAlex, ≥2023	LLM terms AND ("systematic literature review" | "systematic review" | "literature search" | "literature screening") AND ("search" | "screening") — for section 9
L2	OpenAlex, ≥2023	LLM terms AND ("systematic literature review" | "literature review" | "snowballing") AND "software engineering" — for section 9

S3 returned only 1 hit, so the "small scale" concept is covered mainly by snowballing and by the extraction table's model-size column rather than by the search string.

5. Inclusion / exclusion criteria

Include if all of:

• I1 published 2020-01-01 or later (QK-norm was introduced in Oct 2020; nothing earlier is about it). Pre-2020 background (e.g. Vaswani et al. 2017) may be cited, but is not a search result.
• I2 about transformer models, and addresses QK-norm, attention-logit control, or the training stability / LR-sensitivity of transformers.
• I3 English, full text available.

Exclude codes used in search/screening.csv:

Code	Meaning
E1	applies a transformer to another domain (vision, speech, recommendation, medicine…); QK-norm is just one part of the architecture and its effect isn't studied
E2	inference / systems / hardware only (kernels, KV cache, tensor parallelism)
E3	"entropy collapse" in the RL / policy sense, not attention
E4	duplicate, earlier title of an included paper, or a citation fragment rather than a paper
E5	off-topic on title/abstract
E6	(LLM-search set) uses LLMs as the research subject, not as a review tool

Include codes (why it's in):

Code	Meaning
SEED	seed paper
IC	evaluates QK-norm's effect on stability / LR / loss
IA	alternative or complementary stability method, usually compared against QK-norm
IM	mechanism: why logits grow, entropy collapse, attention sinks, massive activations
IS	small-scale controlled ablation study (method precedent)
BG	background / confound (μP, LR transfer, Pre-LN)
LLM	evidence on using LLMs to support literature reviews (section 9)
*	close to the thesis RQ — needs "how my study differs" in the extraction table

6. Snowballing (Wohlin 2014)

1. Backward: every reference of each seed ("what did these guys cite?").
2. Forward: every paper citing each seed.
3. Apply the criteria (I1 automatically; I2 first by keyword filter, then by title/abstract).
4. Each newly included paper becomes a seed for the next iteration. Stop when an iteration adds nothing new.

Status: iteration 1 done for all six seeds (28/10/2026). Iteration 2 not started — [Your call] which of the iteration-1 includes to snowball from (suggest the * ones).

7. Screening procedure and flow (iteration 1)

1. Automatic: de-duplicate by arXiv ID / DOI / title (tools/merge.py).
2. Automatic: drop pre-2020 (I1) and records failing a keyword filter for I2.
3. AI first pass on title/abstract (tools/ai_first_pass.py) → ai_proposal column.
4. Cormac decides → decision column. Disagreements with the AI pass are worth noting — they're evidence for section 9.
5. Full-text read of includes → extraction table.

Stage	QK-norm review	LLM-for-review set
Records retrieved (all searches + snowballing)	2,127 unique, after de-duplication	(included in the 2,127)
Excluded automatically: pre-2020	129	—
Excluded automatically: off-topic keyword filter	1,518 (both sets)
Title/abstract screened	237	243
AI proposal: include / maybe / exclude	33 / 25 / 179	6 / 6 / 231
Your decisions	[pending]	[pending]
Full text read	[pending]	[pending]

8. Data extraction

Template: extraction-table.csv. One row per included paper. Columns: key, citation, year, venue/status, model sizes, data, question asked, main claim, method (what's varied / held fixed), QK-norm isolated? (y/n/what bundled), μP used?, LR sweep (points, range), depth varied?, metrics, close to my RQ (*), how my study differs (≤2 sentences), notes.

Synthesis questions to answer from the table:

• what questions are people asking across the literature?
• which papers are close to the thesis RQ, and for each: two sentences on why the thesis is different (e.g. smaller scale, μP off, single variable, laptop cost, dense sweep vs 3 points)?

9. Use of LLMs in this review (and the paper that backs it)

Supervisor: "if you want to use LLMs for paper search, find a paper that backs up the tactic."

What an LLM (Claude Code) was used for: writing the search scripts; drafting this protocol and the search strings; a first-pass title/abstract screen (suggestions only, kept in a separate column); checking citation metadata against arXiv/Crossref. Not used for: final include decisions, the extraction of claims, or writing thesis text.

Supporting literature (all metadata checked against Crossref or arXiv, 28/10/2026):

• Kitchenham, Pizard, Madeyski, de Souza Santos, Shepperd (2026), Preliminary Guidelines for Using and Evaluating GenAI Tools to Support Systematic Literature Reviews, arXiv:2607.24991. The GUEST recommendations. Conclude GenAI "requires human oversight and is not currently capable of unsupervised systematic studies" but offers "cost-effective assistance for some repetitive tasks". Main citation for the approach taken here (human makes every decision; AI suggests).
• Felizardo et al. (2024), ChatGPT application in SLRs in Software Engineering: an evaluation of its accuracy to support the selection activity, ESEM 2024, doi:10.1145/3674805.3686666.
• Felizardo et al. (2025), On the Difficulties of Conducting and Replicating SLR Studies Using LLMs in Software Engineering, WSESE 2025, doi:10.1109/WSESE66602.2025.00010 — why the search log and fixed scripts matter.
• Wang, Scells, Koopman, Zuccon (2023), Can ChatGPT Write a Good Boolean Query for Systematic Review Literature Search?, SIGIR 2023, doi:10.1145/3539618.3591703 — supports LLM help with search strings.
• Khraisha et al. (2024), Can large language models replace humans in systematic reviews?, Research Synthesis Methods, doi:10.1002/jrsm.1715 — caution: GPT-4 screening performance drops sharply once chance agreement is accounted for.
• Lieberum et al. (2025), Large language models for conducting systematic reviews: on the rise, but not yet ready for use — a scoping review, J. Clinical Epidemiology, doi:10.1016/j.jclinepi.2025.111746.

[Your note: in your own words, why this level of LLM use is defensible given the above.]

10. Deviations log

Date	Planned	Actual	Why
28/10/2026	Semantic Scholar for all searching	Keyword searches via OpenAlex + arXiv; S2 for snowballing only	S2 unauthenticated API returned HTTP 429 on search; OpenAlex lacks arXiv reference lists

11. Defending the review (for the viva / interview)

"This is how I found these papers": six seeds → one round of forward/backward snowballing on Semantic Scholar → keyword searches on arXiv and OpenAlex with logged strings → automatic date and topic filters → title/abstract screening with written reasons → full-text extraction. Every step is a script plus a log, so it can be re-run.