Masters_Thesis/02-literature/search/results/S1-openalex-qknorm.csv

source_id,arxiv,doi,year,title,first_author,n_authors,venue,citations,abstract
W3099096615,,10.18653/v1/2020.findings-emnlp.379,2020,Query-Key Normalization for Transformers,Alex Henry,4,Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP),45,"Low-resource language translation is a challenging but socially valuable NLP task.Building on recent work adapting the Transformer's normalization to this setting, we propose QKNORM, a normalization technique that modifies the attention mechanism to make the softmax function less prone to arbitrary saturation without sacrificing expressivity.Specifically, we apply ℓ 2 normalization along the head dimension of each query and key matrix prior to multiplying them and then scale up by a learnable parameter instead of dividing by the square root of the embedding dimension.We show improvements averaging 0.928 BLEU over state-of-the-art bilingual benchmarks for 5 low-resource translation pairs from the TED Talks corpus and IWSLT'15. 1"
W7164894731,10.48550/arxiv.2606.16310,10.48550/arxiv.2606.16310,2026,QK-Normed MLA: QK normalization without full key caching,Yizhou Han,5,arXiv (Cornell University),0,"Query-key (QK) normalization stabilizes attention by controlling the scale of queries and keys before the dot product, but is not immediately compatible with Multi-head Latent Attention (MLA). MLA achieves efficient decoding by caching low-dimensional latent states instead of full keys, whereas post-projection QK RMSNorm appears to require the fully projected key for every cached token. We show this apparent incompatibility is an implementation artifact, not an architectural constraint. RMSNorm decomposes into a static affine weight and a dynamic scalar RMS statistic. The static key-side weight can be absorbed into the MLA query-side projection; the dynamic key statistic reduces to one inverse-RMS scalar per token and KV group. The resulting formulation is exactly equivalent to explicit post-projection QK RMSNorm in exact arithmetic and preserves MLA's latent decode path. In our 400M runs trained for up to 100B tokens, QK-Normed MLA achieves lower training loss and better downstream accuracy than QK clipping, while H800 decode benchmarks show less than 2% latency overhead up to 256k context. These results make QK normalization a practical stabilization option for MLA models without requiring full-key caching."
W7197026758,,10.1109/lca.2026.3720688,2026,SwiftQK: Fast and Communication-Efficient Tensor Parallelism for Query-Key Normalization,Gyudong Kim,3,IEEE Computer Architecture Letters,0,"Query-Key Normalization (QK-Norm) improves the training stability and quality of modern Large Language Models (LLMs). However, under Tensor Parallelism (TP), layerwise QK-Norm introduces additional cross-GPU communication because the normalization factor depends on the full hidden vector. We present SwiftQK, a multi-GPU RMSNorm kernel that exchanges only scalar normalization statistics and overlaps the remaining Peer-to-Peer reduction with independent element-wise computation in a deadlock-safe persistent kernel. Evaluations on recent LLMs show that SwiftQK reduces QK-Norm latency by 81.4--93.9% relative to the standard TP QK-Norm using full-vector All-Gather. In end-to-end serving, SwiftQK reduces TPOT on average by 29.5% over the All-Gather-based baseline and by 14.3% over an optimized scalar-aggregation implementation."
W4414997173,,10.20533/ijicr.2042.4655.2025.0165,2025,Attention Mechanism Enhancements in Modern Machine Learning,B. Thomas,1,International Journal of Intelligent Computing Research,0,"Attention mechanisms have become a cornerstone of modern machine learning, enabling models (especially Transformers) to weigh the importance of different input elements dynamically.Since the introduction of the Transformer and its Multi-Head Self-Attention (MHA) in 2017, attention-based models have achieved state-of-the-art results in NLP, computer vision, and other domains.However, the flexibility of attention comes at a high computational and memory cost: naïve self-attention requires O(n²) operations and memory with respect to sequence length n.This poses challenges for long sequences and efficient inference, motivating a surge of research into faster and more memory-efficient attention mechanisms.In this article, we review the fundamentals of attention and highlight recent advances aimed at improving attention speed and efficiency, including Multi-Query and Grouped-Query Attention, Rotary Positional Embeddings (RoPE), Query-Key Normalization (QK Norm), hybrid Transformer models, and high-performance implementations like FlashAttention."
W4416789464,10.48550/arxiv.2511.21377,10.48550/arxiv.2511.21377,2025,Controlling changes to attention logits,Ben Anson,2,arXiv (Cornell University),0,"Stability of neural network weights is critical when training transformer models. The query and key weights are particularly problematic, as they tend to grow large without any intervention. Applying normalization to queries and keys, known as `QK norm', fixes stability issues in practice, but is not always applicable. For example, QK norm is not compatible with Multi Latent Attention (MLA) because QK norm requires full materialization of queries and keys during inference, which is not done in MLA. In this paper we suggest that controlling the changes to logits is important for stability. We show that these changes are controllable by assigning parameter-dependent learning rates to the query and key weights. We find that our cheap intervention allows us to increase the base learning rate of the network, outperform other methods in the MLA setting, and achieve performance competitive with QK norm when using Multi-head Attention."
W4407092995,,10.52202/085713-2150,2025,Rope to Nope and Back Again: A New Hybrid Attention Strategy,Bowen Yang,7,neural information processing systems,0,"Long-context large language models (LLMs) have achieved remarkable advancements, driven by techniques like Rotary Position Embedding (RoPE) (Su et al., 2023) and its extensions (Chen et al., 2023; Liu et al., 2024c; Peng et al., 2023). By adjusting RoPE parameters and incorporating training data with extended contexts, we can train performant models with considerably longer input sequences. However, existing RoPE-based methods exhibit performance limitations when applied to extended context lengths. This paper presents a comprehensive analysis of various attention mechanisms, including RoPE, No Positional Embedding (NoPE), and Query-Key Normalization (QK-Norm), identifying their strengths and shortcomings in long-context modeling. Our investigation identifies distinctive attention patterns in these methods and highlights their impact on long-context performance, providing valuable insights for architectural design. Building on these findings, we propose a novel architecture featuring a hybrid attention mechanism that integrates global and local attention spans. This design not only surpasses conventional RoPE-based transformer models with full attention in both long and short context tasks but also delivers substantial efficiency gains during training and inference."
W4395483230,10.48550/arxiv.2404.15702,10.48550/arxiv.2404.15702,2024,Nyonic Technical Report,Junfeng Tian,6,arXiv (Cornell University),0,"This report details the development and key achievements of our latest language model designed for custom large language models. The advancements introduced include a novel Online Data Scheduler that supports flexible training data adjustments and curriculum learning. The model's architecture is fortified with state-of-the-art techniques such as Rotary Positional Embeddings, QK-LayerNorm, and a specially crafted multilingual tokenizer to enhance stability and performance. Moreover, our robust training framework incorporates advanced monitoring and rapid recovery features to ensure optimal efficiency. Our Wonton 7B model has demonstrated competitive performance on a range of multilingual and English benchmarks. Future developments will prioritize narrowing the performance gap with more extensively trained models, thereby enhancing the model's real-world efficacy and adaptability.GitHub: \url{https://github.com/nyonicai/nyonic-public}"
W7165558066,,10.5281/zenodo.20803082,2026,Gating and Normalization Interact in the Attention Block,Shubham Kumar,1,Zenodo (CERN European Organization for Nuclear Research),0,"Abstract We study how an elementwise sigmoid attention-output gate interacts with normalization placement in small causal transformers (29–36M parameters). A 54-model main sweep crosses three schemes (pre-norm; pre-norm + QK-norm; sandwich + QK-norm), three depths, three seeds, and models with and without the gate on WikiText-2. At the best validation checkpoint selected for each run, gating lowers mean validation perplexity in all nine normalization-depth cells. Averaged across depths and seeds, the gated models improve over their matched no-gate baselines by 1.03 PPL with pre-norm, 2.64 with pre-norm + QK-norm, and 3.35 with sandwich + QK-norm. The improvement is therefore 2.32 PPL larger for sandwich + QK-norm than for pre-norm (p < 0.001). A separate fixed-budget 8-layer experiment on WikiText-103 independently finds a gate benefit under all three schemes (all p < 0.01). Normalization placement also produces qualitatively different learned gate distributions: C1 and C2 become low-mean suppressors with essentially no fully-open values, while C3 becomes bimodal, with roughly 20% of values fully closed and 10% fully open. A single-layer ablation identifies the attention-output RMSNorm as necessary and sufficient for this bimodality at similar perplexity, and the distinction persists across the stable learning-rate range. Gate sparsity consistently peaks in the second transformer block. These results show that normalization placement controls both the performance effect and learned geometry of attention-output gating at this scale. Version 2 This version substantially expands the original manuscript with: a fixed-budget 8-layer experiment on WikiText-103; a single-layer ablation isolating the role of the attention-output RMSNorm; learning-rate robustness analyses; a parameter-matched no-gate control; expanded analysis of gate distributions and layer-wise sparsity; additional discussion, limitations, tables, and supplementary results. The experiments use small causal transformers trained from scratch and characterize behaviour in a compute-limited regime. The results should not be assumed to generalise to fully converged or large-scale language-model training without further evidence."
W4414271954,,10.1109/iceict66683.2025.11159965,2025,Enhanced Monocular Depth Estimation Based on Improved Self-Attention Mechanisms and Composite Loss Functions,Mingshen Huang,5,IEEE International Conference on Electronic Information and Communication Technology (ICEICT),0,"Monocular depth estimation, as a key technology in computer vision, has been widely applied in autonomous driving, robotic navigation, and other fields. However, balancing the accuracy of global depth prediction with the fine recovery of local details remains challenging. This paper presents an improved monocular depth estimation framework based on the powerful Depth Anything V2 pre-trained model. First, we fine-tune the ViT-Large pre-trained weights and integrate Query-Key Normalization (QK-Norm), Relative Position Bias (RelPosBias), and Rotary Position Embedding (RoPE) mechanisms into its core self-attention module. These mechanisms enhance the model's ability to perceive spatial information and relative positional relationships, improving the stability and position sensitivity of feature representation. Second, to address common issues of local discontinuities and edge blurring in depth prediction, we construct a composite loss function consisting of improved Scale-invariant Logarithmic Loss (SiLogLoss), Gradient Consistency Loss (GradientLoss), and Edge Smoothness Loss (EdgeSmoothnessLoss), providing supervision from global error, depth map structure, and edge details. We conducted comprehensive experimental evaluations on the KITTI dataset. Results demonstrate that our method achieves significant improvements over the original Depth Anything V2 model in key performance metrics: reducing the Absolute Relative Error (AbsRel) from 0.074 to 0.064, and increasing the $\delta_1$ accuracy from 0.946 to 0.967, validating the effectiveness of our proposed strategies."
W7172429293,10.48550/arxiv.2608.02091,10.48550/arxiv.2608.02091,2026,"One QK Channel, Many Sources: Tracing Low-Precision Attention Collapse",Shuxiao Xie,6,arXiv (Cornell University),0,"A bfloat16 transformer can train normally, then collapse abruptly. Prior work links collapse to structured attention errors and shows QK normalization disrupts their compounding. Distinct low-precision errors trigger the same collapse, leaving unclear whether each needs a fix at its source or one shared route can be blocked instead. We isolated the fault behind a reproduced GPT-2-class collapse to the streaming-softmax accumulator, where an fp32 streaming core repairs it, and turned it into an assay for moving a controlled error across sources. Using it, we found that errors placed outside attention still drove the same QK spectral runaway, and that correcting only QK kept training stable while the fault stayed active. This is a source-channel dissociation: fault source is not failure channel. It held across tested architectures and scales, and reproduced on a second GPU architecture. As a causal probe, projecting each update off the current QK weights' leading three singular directions held the query projection's largest singular value to 11.1, whereas removing equal energy elsewhere left it at 237: the QK channel causally drives the early runaway. What lets the injected error in is temporal sign-coherence, its per-head sign persisting across steps, not aggregate deviation; once inside, the runaway shows as attention-logit saturation. QK-Guard, a dormant controller, tests this by switching on parameter-free QK normalization at the first monitored threshold crossing. On the runs designated for this test, the QK-local action prevented the failure of each matched or same-configuration unguarded run; on plain GPT-2, all 12 final train and validation losses were within 0.03 nat of same-configuration always-on QK-norm, and both methods ran 60k steps without collapse. Intervention at the QK locus therefore suffices in place of a fix at each source."
W7127631375,10.48550/arxiv.2602.02522,10.48550/arxiv.2602.02522,2026,IMU-1: Sample-Efficient Pre-training of Small Language Models,George K. Grigorev,1,arXiv (Cornell University),0,"We present IMU-1, a 430M-parameter language model trained on 72B tokens that approaches the benchmark performance of models trained on 56x more data. We describe a validated training recipe combining recent architectural interventions (QK-norm attention, per-head gating, value residuals, LayerNorm scaling) with optimization advances (NorMuon with cautious weight decay, muP parametrization) and a three-stage training schedule with post-hoc checkpoint EMA. We provide ablations for each component and release code, weights and data to enable reproduction: https://huggingface.co/thepowerfuldeez/imu1_base"
W7214536204,,10.5281/zenodo.23001605,2026,Experiment Receipt Discipline for Small-Model Autoresearch,Reuben Bowlby,1,Zenodo (CERN European Organization for Nuclear Research),0,"Small-model autoresearch loops can generate dozens of experiments per session, but without receipt discipline the results become unauditable: hypotheses blur into outcomes, baselines drift, noise floors go unmeasured, and negative results disappear. This paper presents a case study of the autoresearch-pipeline corpus — 243 result rows, 55 kept patches, 186 DISCARD negative results, 2 crashes — as a worked example of receipt-bearing autoresearch. The strongest finding is not a new architecture patch; it is the governance value of a prospective noise-floor gate that blocks architecture experiments before the noise floor is measured, and the preservation of negative results as first-class evidence. We present the QK-Norm noise-floor-gate vignette as the centerpiece: a case where the noise floor (0.180 untuned) was 36–90× larger than the expected improvement (0.002–0.005 bpb), which the receipt discipline caught before any architecture experiment was authorized."
W4417119521,,10.1109/icassp55912.2026.11463059,2026,DiTSinger: Scaling Singing Voice Synthesis with Diffusion Transformer and Implicit Alignment,Zongcai Du,11,IEEE International Conference on Acoustics Speech and Signal Processing,0,"Recent progress in diffusion-based Singing Voice Synthesis (SVS) demonstrates strong expressiveness but remains limited by data scarcity and model scalability. We introduce a two-stage pipeline: a compact seed set of human-sung recordings is constructed by pairing fixed melodies with diverse LLM-generated lyrics, and melody-specific models are trained to synthesize over 500 hours of high-quality Chinese singing data. Building on this corpus, we propose DiTSinger, a Diffusion Transformer with RoPE and qk-norm, systematically scaled in depth, width, and resolution for enhanced fidelity. Furthermore, we design an implicit alignment mechanism that obviates phoneme-level duration labels by constraining phoneme-to-acoustic attention within character-level spans, thereby improving robustness under noisy or uncertain alignments. Extensive experiments validate that our approach enables scalable, alignment-free, and high-fidelity SVS."
W7135176124,10.48550/arxiv.2603.11611,10.48550/arxiv.2603.11611,2026,"Fractional Rotation, Full Potential? Investigating Performance and Convergence of Partial RoPE",Mohammad Aflah Khan,4,arXiv (Cornell University),0,"Rotary Positional Embedding (RoPE) is widely used in transformers to encode relative positional information, yet the effect of applying RoPE to only a subset of hidden dimensions remains underexplored. This design choice can substantially reduce memory usage, especially at long context lengths. We show that partial RoPE achieves up to 10$\times$ memory savings over standard RoPE cache while maintaining comparable final loss. We systematically study its impact on training dynamics and convergence across architectures, model sizes, sequence lengths, and datasets. Our results reveal that: (1) applying RoPE to only a small fraction of dimensions (around 10\%) matches the convergence of full RoPE; (2) these trends remain consistent across settings; and (3) NoPE (No Positional Encoding) can produce loss spikes, which are mitigated by minimal RoPE usage or QK-Norm application. Overall, our findings highlight partial RoPE as an effective trade-off between efficiency and stability."
W7133339030,10.48550/arxiv.2603.02170,10.48550/arxiv.2603.02170,2026,SageBwd: A Trainable Low-bit Attention,Jintao Zhang,8,arXiv (Cornell University),0,"Low-bit attention, such as SageAttention, has emerged as an effective approach for accelerating model inference, but its applicability to training remains poorly understood. In prior work, we introduced SageBwd, a trainable INT8 attention that quantizes six of seven attention matrix multiplications while preserving fine-tuning performance. However, SageBwd exhibited a persistent performance gap to full-precision attention (FPA) during pre-training. In this work, we investigate why this gap occurs and demonstrate that SageBwd matches full-precision attention during pretraining. Through experiments and theoretical analysis, we reach a few important insights and conclusions: (i) QK-norm is necessary for stable training at large tokens per step, (ii) quantization errors primarily arise from the backward-pass score gradient dS, (iii) reducing tokens per step enables SageBwd to match FPA performance in pre-training, and (iv) K-smoothing remains essential for training stability, while Q-smoothing provides limited benefit during pre-training."
W7128556445,,,2026,OneLive: Dynamically Unified Generative Framework for Live-Streaming Recommendation,Shen Wang,18,arXiv (Cornell University),0,"Live-streaming recommender system serves as critical infrastructure that bridges the patterns of real-time interactions between users and authors. Similar to traditional industrial recommender systems, live-streaming recommendation also relies on cascade architectures to support large-scale concurrency. Recent advances in generative recommendation unify the multi-stage recommendation process with Transformer-based architectures, offering improved scalability and higher computational efficiency. However, the inherent complexity of live-streaming prevents the direct transfer of these methods to live-streaming scenario, where continuously evolving content, limited lifecycles, strict real-time constraints, and heterogeneous multi-objectives introduce unique challenges that invalidate static tokenization and conventional model framework. To address these issues, we propose OneLive, a dynamically unified generative recommendation framework tailored for live-streaming scenario. OneLive integrates four key components: (i) A Dynamic Tokenizer that continuously encodes evolving real-time live content fused with behavior signal through residual quantization; (ii) A Time-Aware Gated Attention mechanism that explicitly models temporal dynamics for timely decision making; (iii) An efficient decoder-only generative architecture enhanced with Sequential MTP and QK Norm for stable training and accelerated inference; (iv) A Unified Multi-Objective Alignment Framework reinforces policy optimization for personalized preferences."
W7128476908,10.48550/arxiv.2602.08612,10.48550/arxiv.2602.08612,2026,OneLive: Dynamically Unified Generative Framework for Live-Streaming Recommendation,Shen Wang,18,arXiv (Cornell University),0,"Live-streaming recommender system serves as critical infrastructure that bridges the patterns of real-time interactions between users and authors. Similar to traditional industrial recommender systems, live-streaming recommendation also relies on cascade architectures to support large-scale concurrency. Recent advances in generative recommendation unify the multi-stage recommendation process with Transformer-based architectures, offering improved scalability and higher computational efficiency. However, the inherent complexity of live-streaming prevents the direct transfer of these methods to live-streaming scenario, where continuously evolving content, limited lifecycles, strict real-time constraints, and heterogeneous multi-objectives introduce unique challenges that invalidate static tokenization and conventional model framework. To address these issues, we propose OneLive, a dynamically unified generative recommendation framework tailored for live-streaming scenario. OneLive integrates four key components: (i) A Dynamic Tokenizer that continuously encodes evolving real-time live content fused with behavior signal through residual quantization; (ii) A Time-Aware Gated Attention mechanism that explicitly models temporal dynamics for timely decision making; (iii) An efficient decoder-only generative architecture enhanced with Sequential MTP and QK Norm for stable training and accelerated inference; (iv) A Unified Multi-Objective Alignment Framework reinforces policy optimization for personalized preferences."
W4415310639,10.48550/arxiv.2507.13338,10.48550/arxiv.2507.13338,2025,Training Transformers with Enforced Lipschitz Constants,Laker Newhouse,6,arXiv (Cornell University),1,"Neural networks are often highly sensitive to input and weight perturbations. This sensitivity has been linked to pathologies such as vulnerability to adversarial examples, divergent training, and overfitting. To combat these problems, past research has looked at building neural networks entirely from Lipschitz components. However, these techniques have not matured to the point where researchers have trained a modern architecture such as a transformer with a Lipschitz certificate enforced beyond initialization. To explore this gap, we begin by developing and benchmarking novel, computationally-efficient tools for maintaining norm-constrained weight matrices. Applying these tools, we are able to train transformer models with Lipschitz bounds enforced throughout training. We find that optimizer dynamics matter: switching from AdamW to Muon improves standard methods -- weight decay and spectral normalization -- allowing models to reach equal performance with a lower Lipschitz bound. Inspired by Muon's update having a fixed spectral norm, we co-design a weight constraint method that improves the Lipschitz vs. performance tradeoff on MLPs and 2M parameter transformers. Our 2-Lipschitz transformer on Shakespeare text reaches validation accuracy 60%. Scaling to 145M parameters, our 10-Lipschitz transformer reaches 21% accuracy on internet text. However, to match the NanoGPT baseline validation accuracy of 39.4%, our Lipschitz upper bound increases to 10^264. Nonetheless, our Lipschitz transformers train without stability measures such as layer norm, QK norm, and logit tanh softcapping."
W7221787881,10.48550/arxiv.2610.17969,10.48550/arxiv.2610.17969,2026,Neural Dimensional Analysis: Foundations of Dimensionless and Scale-Consistent Computation,Bum Jun Kim,4,arXiv (Cornell University),0,"Modern Transformers rely on many scale-control mechanisms, including normalization, residual scaling, attention temperatures, loss weighting, and adaptive updates. To explain how these mechanisms maintain scale consistency, we introduce Neural Dimensional Analysis (NDA), a unified framework based on dimensional consistency. We define neural units that specify how tensors rescale and which quantities can be added, just as physical units do in physics. Our rules match residual units, nondimensionalize attention and gate inputs, combine losses across units, and convert gradient units into parameter units. We implement NDA through dimensionless Unit Violation Indices (UVI) for diagnosing scale mismatches within Transformers, a unit-consistent multi-loss objective, and the Gauge-Canonical Transformer (GCT) with normalized coordinates and orthogonalized residual additions. In language-model experiments spanning 51M to 1B parameters, GCT remains stable during training across a broad sixteenfold learning-rate range, outperforming Pre-LN, LLaMA-style, and QK-Norm in 124M validation loss at tuned learning rates. GCT's normalized residual stream also shows improved robustness to quantization. At 1B parameters, calibrated per-tensor int8 quantization increases GCT's validation loss by only 0.009 nats, compared with up to 5.51 nats for the baselines. Vision and physics-informed neural network experiments further confirm NDA's benefits beyond language modeling. These results establish dimensional consistency as a neural network design principle."
W7213292946,10.48550/arxiv.2609.14715,10.48550/arxiv.2609.14715,2026,Depth and Scale in the Sub-150M Regime: JugnuLM-53M vs JugnuLM-110M,Dushyant Rajput,3,arXiv (Cornell University),0,"We scale our conventional sub-150M pretraining recipe from 53.5M to 109.7M parameters, holding the method fixed (Qwen3-style decoder with grouped-query attention, RoPE, SwiGLU, RMSNorm, QK-Norm, and a z-loss; FineWeb-Edu data) and changing only the geometry to a deep-and-thin 23-layer x 576-hidden design. The larger model improves across the board -- BLiMP 78.1 -> 81.3, ARC-Easy 51.4 -> 52.5, WikiText-2 byte-perplexity 2.04 -> 1.95 -- and its 81.3% BLiMP essentially matches GPT-X2-125M (81.28) at about 12% fewer parameters. Notably the 110M model achieves this on fewer training tokens (about 8B vs 12B), so the gain is attributable to capacity and depth, not more data. Both models are deliberately conventional; this report is a clean scaling control and the baseline rung (R0) of an ablation study of what further improves models in this regime. An ablation ladder follows: value residuals (R1) and the Muon optimizer (R2) lift ARC-Easy by a cumulative +3.6 (52.5 -> 56.1) at a near-flat BLiMP and are kept; a diverse data blend (R3) and two logit-distillation settings (R4a/R4b) are not kept -- honest negatives. R3 pins ARC-Easy to FineWeb-Edu's educational filtering rather than raw diversity; distillation from a 1.7B teacher can reach the class-leading ARC-Easy (56.99, matching GPT-X2-125M) but only at a perplexity cost that dialing KD down then erases -- so R2 remains the best kept stack."
W7134257203,,10.5281/zenodo.18924690,2026,Modernizing GPT-2-Scale Models Under a 10B-Token Training Budget,Hang Lu,1,Zenodo (CERN European Organization for Nuclear Research),0,"This technical report investigates whether modern architectural and optimization strategies improve language modeling performance at the GPT-2 (124M) scale under a strict 10B-token single-pass training budget. Starting from a GPT-2-style configuration, the study introduces Rotary Positional Embeddings (RoPE), SwiGLU activation, RMSNorm, Query-Key normalization, and a hybrid Muon + AdamW optimization scheme while keeping the parameter count fixed. All models are trained under the same 10B-token single-pass setting. The report also includes a controlled learning-rate schedule ablation (Warmup–Plateau–Cosine vs. Warmup–Cosine) and analyzes both final metrics and training dynamics. Results suggest that the modernized GPT-2-scale recipe improves validation efficiency under a fixed token budget, while downstream gains remain modest and schedule-dependent. This upload contains the public PDF version of the paper."
W7160755163,,10.5281/zenodo.20114347,2026,VectraYX-Nano: A 42M-Parameter Spanish Cybersecurity LLM with Native MCP Tool Integration,Juan Salas Santillana,1,Zenodo (CERN European Organization for Nuclear Research),0,"We present VectraYX-Nano, a 41.95M-parameter decoder-onlylanguage model trained from scratch in Spanish for cybersecurity,with a Latin-American regional focus and native tool invocationvia the Model Context Protocol (MCP). The model is built aroundfour contributions. (i) Corpus. VectraYX-Sec-ES, a 170M-tokenSpanish corpus assembled by an eight-VM distributed pipeline at∼$25 USD of cloud compute and partitioned into three curriculumphases: conversational (42M tokens, OpenSubtitles-ES [35] andOASST1 [33]), cybersecurity (118M tokens, NVD [39], Wikipedia-ES, in-house NVD-derived Spanish CVE mirror, security blogs),and offensive-security tooling (10M tokens, ExploitDB, HackTricks,OWASP). (ii) Architecture. A 42M-parameter Transformerdecoder combining Grouped-Query Attention [2], QK-Norm [14],RMSNorm [58], SwiGLU [49], RoPE [52], and a 𝑧-loss auxiliary [11],paired with a domain-balanced 16,384-token byte-fallbackBPE [34, 48] trained on a 50/50 conversational/technical mix-ture. (iii) Curriculum with replay. Continual pre-trainingacross the three phases with a replay buffer [30] mitigatescatastrophic forgetting [19, 32] and yields a monotonic lossdescent (9.80 → 3.17 → 3.00 → 2.16). After SFT (final loss 1.74)on a curriculum-aware mixture of OASST-ES, Alpaca-ES, CVEQ&A, and tool-use traces, the v2 bootstrap-ablation referenceattains a conversational gate of 0.775 ± 0.043 on B5 over 𝑁 = 4seeds (Section 8.6), and a controlled Phase-2 replay sweep over{0, 5, 10, 25, 50}% saturates B5 at ≥ 25% replay (Section 6.8).(iv) Two empirical findings, both N=4. A controlled bootstrap-corpus ablation across v2 (OpenSubs), v4 (mC4-ES [57]), andv6 (60/25/15 OpenSubs/mC4/Wiki) under 𝑁 = 4 seeds exposesa loss-versus-register inversion: the lower-perplexity bootstrapsyield measurably worse conversational behavior (𝑣2 > 𝑣4 > 𝑣6on B5 at every paired seed), indicating that at the nano scale thebootstrap-corpus register dominates downstream chat quality.The B4 (tool-selection) floor of 0.000 on the mixed SFT corpusis a corpus-density artifact, not a capacity gate: rebalancing theSFT mixture to tool-use ratio 1:21 yields VectraYX-Nano v7, thereleased headline configuration, which reaches B4 = 0.230 ± 0.052at 42M parameters (𝑁 = 4 seeds) while retaining B1 = 0.332 ± 0.005and B5 = 0.725 ± 0.130 (Section 8.7); a LoRA [29] replication on a260M from-scratch mid-tier reaches 0.445 ± 0.201. The releasedGGUF [22] artifact is 81 MB in F16 (approximately 20 MB in4-bit quantization), runs at sub-second time-to-first-token oncommodity hardware under llama.cpp [21], and is, to the best ofour knowledge, the first published Spanish-native cybersecurityLLM with end-to-end MCP integration. We release the corpusconstruction recipe, training scripts, configurations, GGUF weights,∗ The author is employed at Globant. Institutional affiliation approval is pending.and the B1–B5 benchmark suite (B1: 500, B2: 200, B3: 100, B4: 200,B5: 314 prompts) for reproducibility."
W7112695331,,,2025,"Duality, Weight Decay, and Metrized Deep Learning","Newhouse, Laker",1,DSpace@MIT (Massachusetts Institute of Technology),0,"The Muon optimizer has shown convincing evidence that it is faster and more scalable than AdamW for deep learning training, setting speed records for training NanoGPT and scaling up to models with 16B parameters. The theory that led to Muon is called metrized deep learning, a method that suggests assigning norms to each part of a neural network. Chapter 1 begins with an accessible explanation of metrized deep learning, including one of its recurring tools: odd polynomial iterations that act directly on singular values. Chapter 2 reviews duality, a way to modify the gradient that seeks to decrease the loss the most while disturbing the model the least. Pedagogically, duality links four popular optimizers—SGD, Adam, Shampoo, and Muon—under a common framework, steepest descent under a norm. Practically, experiments suggest that duality-based optimizers train faster than AdamW and transfer learning rate across width. Chapter 3 develops tools to enforce weight norm constraints during training, conferring provable and upfront Lipschitz guarantees for transformers. We find that optimizer dynamics matter: switching from AdamW to Muon improves standard weight regularization methods—weight decay and spectral normalization—allowing models to reach equal performance with a lower Lipschitz bound. Leveraging that Muon’s update has a fixed spectral norm, we co-design a weight constraint method called spectral cap that improves the Lipschitz vs. performance tradeoff for MLPs and 2M parameter transformers. Our 4-Lipschitz transformer on Shakespeare text reaches validation accuracy 60%. Scaling to 145M parameters, our 600-Lipschitz transformer reaches 21% accuracy on internet text. However, to match the NanoGPT baseline validation accuracy of 39.4%, our Lipschitz upper bound increases to 10^274. Nonetheless, our Lipschitz transformers train without stability measures such as layer norm, QK norm, and tanh logit softcapping."
W7162043968,,,2026,A Two-Parameter Weibull Framework for Diagnosing Transformer Weight Distributions,Tiexin Ding,1,arXiv (Cornell University),0,"We apply the Weibull distribution -- a two-parameter family from extreme-value theory -- as a diagnostic framework for element-wise weight magnitude distributions in transformers. At initialization, i.i.d. Gaussian weights give |w| ~ HalfNormal, yielding k ~ 1.20 via middle-80% probability-plot fit (the protocol used throughout this work). This anchor makes k a principled, architecture-independent measuring stick for training dynamics; fitting each weight matrix independently at every layer at every checkpoint enables per-component, per-layer, and per-step diagnostics that aggregate statistics cannot resolve. Applying this framework to 12 model entries spanning 7 architectural families (Pythia, OLMo-1/2, LLaMA-3, Mistral, Qwen2.5/3) reveals three findings. First, FFN modules and the attention output projection W_o -- the Transmission Class -- fall in a narrow k band: median terminal k in [1.186, 1.204] across 12 entries (cross-family CV = 0.51%), shared across SwiGLU/GeLU activations, Pre-LN/QK-Norm placements, and 70M-14B sizes. Second, the attention input projections W_q, W_k -- the Selection Class -- depart from the Weibull family, with severity shaped by storage: separately-stored Q/K (OLMo-1, OLMo-2) yields k in [0.76, 0.99] (deep); GQA models yield k in [1.10, 1.16] (mild); Pythia's merged W_qkv occupies a transitional zone tracking training budget T/tau monotonically. Third, lambda grows substantially during training and scales with sqrt(eta/lambda_wd) within the Pythia family (Pearson r = 0.94, three Transmission kinds), directionally consistent with Fan et al. (2025). The two parameters carry independent information: k labels the functional class, lambda labels training progress. We release npm-weibull-py v0.4 (Python library) and DATABASE_v9_1 at https://github.com/tiexinding/NPM-Weibull-public ."
W7161786876,10.48550/arxiv.2605.18898,10.48550/arxiv.2605.18898,2026,A Two-Parameter Weibull Framework for Diagnosing Transformer Weight Distributions,Tiexin Ding,1,arXiv (Cornell University),0,"We apply the Weibull distribution -- a two-parameter family from extreme-value theory -- as a diagnostic framework for element-wise weight magnitude distributions in transformers. At initialization, i.i.d. Gaussian weights give |w| ~ HalfNormal, yielding k ~ 1.20 via middle-80% probability-plot fit (the protocol used throughout this work). This anchor makes k a principled, architecture-independent measuring stick for training dynamics; fitting each weight matrix independently at every layer at every checkpoint enables per-component, per-layer, and per-step diagnostics that aggregate statistics cannot resolve. Applying this framework to 12 model entries spanning 7 architectural families (Pythia, OLMo-1/2, LLaMA-3, Mistral, Qwen2.5/3) reveals three findings. First, FFN modules and the attention output projection W_o -- the Transmission Class -- fall in a narrow k band: median terminal k in [1.186, 1.204] across 12 entries (cross-family CV = 0.51%), shared across SwiGLU/GeLU activations, Pre-LN/QK-Norm placements, and 70M-14B sizes. Second, the attention input projections W_q, W_k -- the Selection Class -- depart from the Weibull family, with severity shaped by storage: separately-stored Q/K (OLMo-1, OLMo-2) yields k in [0.76, 0.99] (deep); GQA models yield k in [1.10, 1.16] (mild); Pythia's merged W_qkv occupies a transitional zone tracking training budget T/tau monotonically. Third, lambda grows substantially during training and scales with sqrt(eta/lambda_wd) within the Pythia family (Pearson r = 0.94, three Transmission kinds), directionally consistent with Fan et al. (2025). The two parameters carry independent information: k labels the functional class, lambda labels training progress. We release npm-weibull-py v0.4 (Python library) and DATABASE_v9_1 at https://github.com/tiexinding/NPM-Weibull-public ."
W7151240155,,10.1109/ieeeconf68869.2026.11461110,2026,Efficient Russian-English Neural Machine Translation with Architectural Optimizations,German Berezin,2,Systems of Signals Generating and Processing in the Field of on Board Communications,0,"This work addresses efficient neural machine translation deployment for Russian-English translation in resource-constrained onboard communication environments. We propose an enhanced Transformer architecture integrating three key optimizations: Root Mean Square Normalization (RMSNorm) for reduced computational overhead, Query-Key normalization (QKNorm) for robust attention mechanisms, and SwiGLU activation functions for improved parameter efficiency. The approach is evaluated on the OPUS 100 Russian-English parallel corpus using BLEU scores as the primary evaluation metric. The proposed 32-million-parameter model achieves 11.05 BLEU, representing an 8.07 BLEU point improvement over the baseline Transformer with identical parameter count, and demonstrates 3.71x higher parameter efficiency. Ablation studies indicate that architectural design choices significantly impact translation performance within fixed computational budgets. The enhanced architecture enables efficient neural translation deployment on resource-constrained aerospace, maritime, and satellite communication platforms. This work presents a novel architectural combination specifically optimized for Russian-English translation, offering practical insights for deployment of neural translation systems in specialized computing environments."
W7161249368,10.48550/arxiv.2605.13989,10.48550/arxiv.2605.13989,2026,VectraYX-Nano: A 42M-Parameter Spanish Cybersecurity Language Model with Curriculum Learning and Native Tool Use,Juan S. Santillana,1,arXiv (Cornell University),0,"We present VectraYX-Nano, a 41.95M-parameter decoder-only language model trained from scratch in Spanish for cybersecurity, with a Latin-American regional focus and native tool invocation via the Model Context Protocol (MCP). The model has four contributions. (i) Corpus: VectraYX-Sec-ES, a 170M-token Spanish corpus assembled by an eight-VM distributed pipeline at ~$25 USD of cloud compute and split into three curriculum phases (conversational 42M, cybersecurity 118M, offensive tooling 10M). (ii) Architecture: a 42M Transformer decoder with GQA, QK-Norm, RMSNorm, SwiGLU, RoPE and z-loss, paired with a domain-balanced 16,384-token byte-fallback BPE. (iii) Curriculum with replay across the three phases yields a monotonic loss descent (9.80 -> 3.17 -> 3.00 -> 2.16); after SFT (loss 1.74) the v2 bootstrap-ablation reference attains a conversational gate of 0.775 +/- 0.043 on B5 over N=4 seeds, and a controlled Phase-2 replay sweep over {0,5,10,25,50}% saturates B5 at >=25% replay. (iv) Two empirical findings, both N=4. A controlled bootstrap-corpus ablation across v2 (OpenSubs), v4 (mC4-ES), and v6 (60/25/15 OpenSubs/mC4/Wiki) exposes a loss-versus-register inversion: lower-perplexity bootstraps yield measurably worse conversational behavior (v2 > v4 > v6 on B5 at every paired seed). The B4 (tool-selection) floor of 0.000 is a corpus-density artifact, not a capacity gate: rebalancing the SFT mixture to tool-use ratio 1:21 yields VectraYX-Nano v7, the released headline configuration, reaching B4 = 0.230 +/- 0.052 at 42M while retaining B1 = 0.332 +/- 0.005 and B5 = 0.725 +/- 0.130; a LoRA replication on a 260M from-scratch mid-tier reaches 0.445 +/- 0.201. The released GGUF is 96 MB in F16, runs sub-second TTFT on commodity hardware under llama.cpp, and is, to our knowledge, the first published Spanish-native cybersecurity LLM with end-to-end MCP integration."
W7163591057,10.48550/arxiv.2606.04405,10.48550/arxiv.2606.04405,2026,Low-Rank Decay for Grokking in Scale-Invariant Transformers: A Spectral-Geometric View,Mingyu Li,1,arXiv (Cornell University),0,"Modern Transformer architectures frequently employ normalization mechanisms such as RMSNorm and Query-Key Normalization, making parts of the model approximately scale-invariant with respect to weight magnitudes. In this regime, standard Frobenius-norm weight decay acts purely along the radial direction of the weight space and cannot directly simplify the function represented by the normalized layer. We study grokking in small algorithmic tasks through this lens and propose \emph{Low-Rank Decay} (LRD), a nuclear-norm-like spectral regularizer whose subgradient -- the polar factor $UV^\top$ -- retains a tangential component even in the scale-invariant setting. This distinction has a concrete dynamical consequence: after the model memorizes the training set and task gradients vanish, L2 decay can no longer reshape the weight spectrum, whereas LRD continues to compress singular values in an $\ell_1$-like fashion. On modular arithmetic tasks, we find that LRD induces rapid effective-rank collapse in Query/Key matrices and expands the data-fraction boundary at which delayed generalization (grokking) occurs. We further provide a spectral-geometric interpretation through the ``needle-to-fan'' expansion of the nuclear-norm subdifferential near low-rank strata."
W7196970926,,10.52202/085713-2786,2025,"Attention Sinks: A 'Catch, Tag, Release' Mechanism for Embeddings",Stephen Zhang,3,neural information processing systems,0,"Large language models (LLMs) often concentrate their attention on a few specific tokens referred to as attention sinks. Common examples include the first token, a prompt-independent sink, and punctuation tokens, which are prompt-dependent. While the tokens causing the sinks often lack direct semantic meaning, the presence of the sinks is critical for model performance, particularly under model compression and KV-caching. Despite their ubiquity, the function, semantic role, and origin of attention sinks -- especially those beyond the first token -- remain poorly understood. In this work, we conduct a comprehensive investigation demonstrating that attention sinks: catch a sequence of tokens, tag them using a common direction in embedding space, and release them back into the residual stream, where tokens are later retrieved based on the tags they have acquired. Probing experiments reveal these tags carry semantically meaningful information, such as the truth of a statement. These findings extend to reasoning models, where the mechanism spans more heads and explains greater variance in embeddings, or recent models with query-key normalization, where sinks remain just as prevalent. To encourage future theoretical analysis, we introduce a minimal problem which can be solved through the 'catch, tag, release' mechanism, and where it emerges through training."
W4406755454,10.48550/arxiv.2501.12352,10.48550/arxiv.2501.12352,2025,Test-time regression: a unifying framework for designing sequence models with associative memory,Ke Alexander Wang,3,arXiv (Cornell University),0,"Sequence models lie at the heart of modern deep learning. However, rapid advancements have produced a diversity of seemingly unrelated architectures, such as Transformers and recurrent alternatives. In this paper, we introduce a unifying framework to understand and derive these sequence models, inspired by the empirical importance of associative recall, the capability to retrieve contextually relevant tokens. We formalize associative recall as a two-step process, memorization and retrieval, casting memorization as a regression problem. Layers that combine these two steps perform associative recall via ``test-time regression'' over its input tokens. Prominent layers, including linear attention, state-space models, fast-weight programmers, online learners, and softmax attention, arise as special cases defined by three design choices: the regression weights, the regressor function class, and the test-time optimization algorithm. Our approach clarifies how linear attention fails to capture inter-token correlations and offers a mathematical justification for the empirical effectiveness of query-key normalization in softmax attention. Further, it illuminates unexplored regions within the design space, which we use to derive novel higher-order generalizations of softmax attention. Beyond unification, our work bridges sequence modeling with classic regression methods, a field with extensive literature, paving the way for developing more powerful and theoretically principled architectures."
W7222457667,10.48550/arxiv.2610.22330,10.48550/arxiv.2610.22330,2026,Designing Transformers for Continuous Diffusion Language Models,Weijian Luo,5,arXiv (Cornell University),0,"Continuous diffusion language models (CDLMs) offer an alternative approach to parallel language generation; however, most existing studies directly use transformer architectures from either causal large language models (LLMs) or Diffusion Transformers (DiTs) for image generation, which were not optimized for CDLMs. In this paper, we conduct a systematic study on Transformer architectures for CDLMs and introduce Language Diffusion Transformer (LangDiT), combining feature and query-key normalization, wider noise conditioning, likelihood-shortcut gradient routing and calibration, and confidence-gated training self-conditioning. LangDiT significantly reduces generation perplexity in matched LangFlow comparisons, establishing a new state of the art in continuous language generation across different model formulations. Using LangFlow as a representative CDLM, LangDiT as a plug-and-play backbone leads to a new state-of-the-art model with a generation perplexity (GenPPL) of 21.72 using 1024 NFEs and 44.72 GenPPL using 128 NFEs on the unconditional generation of 1024-token sequences on the OpenWebText dataset. When adapted to Plaid-style CDLM formulations, LangDiT results in a new leading GenPPL of 11.33 using 1024 denoising NFEs, and 27.51 GenPPL using 128 denoising NFEs. These gains are achieved while retaining each model's underlying representation, training objective, and inference sampler, isolating transformer architecture as a fundamental source of improvement. Our study identifies architectural design as an important source of improvement in continuous diffusion language models."
W4403364498,10.48550/arxiv.2410.07563,10.48550/arxiv.2410.07563,2024,PLaMo-100B: A Ground-Up Language Model Designed for Japanese Proficiency,"Elements, Preferred",21,arXiv (Cornell University),0,"We introduce PLaMo-100B, a large-scale language model designed for Japanese proficiency. The model was trained from scratch using 2 trillion tokens, with architecture such as QK Normalization and Z-Loss to ensure training stability during the training process. Post-training techniques, including Supervised Fine-Tuning and Direct Preference Optimization, were applied to refine the model's performance. Benchmark evaluations suggest that PLaMo-100B performs well, particularly in Japanese-specific tasks, achieving results that are competitive with frontier models like GPT-4. The base model is available at https://huggingface.co/pfnet/plamo-100b."
W7223737153,,10.5281/zenodo.23179058,2026,"Rounding error in FlashAttention: centered, uncentered, and radial",Hanyu Yang,1,Zenodo (CERN European Organization for Nuclear Research),0,"FlashAttention computes attention one block of keys at a time and divides by the softmax normalizer only at the end. How large is its rounding error, and is it worse than that of ordinary two-pass attention? This note bounds the forward error of FlashAttention-2 for one query, compares it with two-pass attention, and measures the one place where it really goes wrong: the split-KV (Flash-Decoding) merge. Main results One identity does most of the work. An error that hits a key's weight in both the numerator and the normalizer costs only the spread of the values around the output, not their size. Most of FlashAttention's roundings are of this kind. No worst-case winner. FlashAttention-2 and two-pass attention share the same leading term: the cast of the weights to bf16 or fp16. Peaked attention favours FlashAttention-2. The key at the running maximum has weight exactly 1 and casts without error; two-pass casts 1/ℓ, which is rarely representable. The split-KV merge breaks the structure. Its weights are never renormalized, so rounding the global log-sum-exp scales the whole output, and the error grows with the logits. Renormalizing the merge removes it; FlashAttention-3, FlashInfer, vLLM and xformers already merge this way. A published bound fails. The O(u log n) bound of Hsu et al. (ELSA, Theorem 1) is false as stated; Section 6 gives counterexamples from cancelling values and from large logits. Measured on an NVIDIA L4 (PyTorch 2.11 FlashAttention-2, fp16, d = 64, τ = 1/8, identical keys, all values 1, so the exact output is 1). Largest |ŷ − 1| in each logit range, in units of 2−11: batch × keyskernelτs < 27[27, 212)[212, 216)τs ≥ 216 24 × 256single pass0000 24 × 1024split-KV, T = 400≤ 1≤ 24 24 × 2048split-KV, T = 400≤ 1≤ 24 1 × 4096split-KV, T = 1600≤ 2≤ 46 The worst case, 46 × 2−11, is a 2.2% error on an output that should be exactly 1. The figure file plots all 160 logit levels per shape; the CSV holds the raw measurements. Open question. The split-KV error survives the final fp16 cast only once τ max s reaches the thousands. Do models without QK-norm, or models whose logits grow late in training, reach that range during decoding? Also on alphaXiv: the paper and an illustrated companion post. Files. note.pdf (the paper, with full proofs in the appendix); fig_l4_splitkv_error.png (the measurements above, per logit level); l4_splitkv_levels.csv (640 rows: shape, logit level, measured error, and the bound where it applies)."
W7211947725,,10.1007/978-3-032-30438-4_3,2026,Developing GlassGPT: A Modern Day GPT-2 Alternative,Arjun G. Ravi,4,Lecture notes in networks and systems,0,
W7149007078,10.48550/arxiv.2604.01683,10.48550/arxiv.2604.01683,2026,Invertible Query-Key Coupling Composes with Attention Mechanisms,Barak Gahtan,2,arXiv (Cornell University),0,"Scaled dot-product attention forms its queries and keys as independent linear projections, so the two never interact before the dot product that scores them. We study coupled query-key dynamics, a pre-scoring transformation that evolves each token's query and key jointly through a shared invertible coupling before standard scoring. We realize it as an alternating affine map in the style of real non-volume-preserving flows: the coupling is the identity at initialization, adds a small fraction of parameters per head, and leaves the softmax and surrounding architecture unchanged. We place coupling on top of existing attention methods rather than replacing them, and ask whether that composition helps. On WikiText-103, adding coupling to Differential Attention improves on it at both 150M and 455M parameters. At 455M the gain is significant at sequence length 512 (p=0.003, six seeds), survives a Bonferroni correction and replicates on a held-out test split; it also holds across rotary-embedding training lengths 512, 1024, and 2048. The same additive direction appears when coupling is added to query-key normalization (significant at 150M) and Multi-Token Attention (directional). Matched controls attribute the gain to the joint pre-scoring coupling rather than to added capacity, and show that removing the invertibility guarantee preserves the 455M gain yet is far worse than the base method at 150M, so invertibility is what makes the coupling reliable across scales. On its own, coupling lowers perplexity at 60M and 150M (one-sided Welch tests, p<0.05) but the gain narrows with scale and is not significant at 455M. We relate the construction to the expressivity of coupling flows, use an associative-recall study to map where coupling helps and where it degrades sharp retrieval, and conclude that coupling is most useful in composition with a scoring-stage method rather than as a standalone change."
W4414848277,10.48550/arxiv.2502.17092,10.48550/arxiv.2502.17092,2025,Shakti-VLMs: Scalable Vision-Language Models for Enterprise AI,Syed Abdul Gaffar Shakhadri,3,arXiv (Cornell University),0,"We introduce Shakti VLM, a family of vision-language models in the capacity of 1B and 4B parameters designed to address data efficiency challenges in multimodal learning. While recent VLMs achieve strong performance through extensive training data, Shakti models leverage architectural innovations to attain competitive results with fewer tokens. Key advancements include QK-Normalization for attention stability, hybrid normalization techniques, and enhanced positional encoding. A three-stage training strategy further optimizes learning efficiency. Evaluations show that Shakti-Shakti-VLM-1B and Shakti-VLM-4B excel in document understanding, Visual Reasoning, OCR extraction, and general multimodal reasoning. Our results highlight that high performance can be achieved through model design and training strategy rather than sheer data volume, making Shakti an efficient solution for enterprise-scale multimodal tasks."
W7124649764,,10.5281/zenodo.18293025,2026,Complexity-Deep: Token-Routed MLP with Mu-Guided Dynamics for Efficient Transformer Architectures,Boris Peyriguere,1,Zenodo (CERN European Organization for Nuclear Research),0,"We present Complexity-Deep, a transformer architecture introducing two key innovations: (1) Token-Routed MLP, a deterministic expert routing mechanism based on token identity that achieves perfect load balancing without auxiliary losses, and (2) Mu-Guided Dynamics, a simplified PID-inspired system that accumulates context across layers and influences both attention and expert routing. Our Token-Routed MLP uses modulo-based routing (expert_id = token_id mod N) ensuring uniform expert utilization regardless of token frequency distribution. We further introduce CGGR (Contiguous Group GEMM Routing), a Triton-based kernel optimization achieving 5-6x speedup over batched matrix multiplication. The architecture incorporates modern techniques including Grouped Query Attention (GQA), RoPE positional embeddings, QK Normalization, and Flash Attention via SDPA. Our 1.5B parameter model demonstrates the viability of deterministic routing as an alternative to learned routing in Mixture-of-Experts architectures."
W7212826526,10.48550/arxiv.2609.12330,10.48550/arxiv.2609.12330,2026,Unleashing the Power of Equality Saturation for Tensor Program Superoptimization,Qi Zhan,4,arXiv (Cornell University),0,"Efficient GPU implementations of tensor programs often require joint optimization of high-level algebraic formulations and low-level execution strategies. However, the resulting search space grows rapidly as transformations combine across operators, making joint optimization difficult to scale. We present EqiForge, a tensor program superoptimizer based on equality saturation. Its unified IR represents high-level tensor expressions and tiled computations in a single expression language. By composing equality rules, EqiForge derives fused implementations such as FlashAttention-style kernels directly from tensor expressions. Early compaction prunes redundant partial programs before completion, while subgraph composition extends the search to larger graphs. Across tensor-program benchmarks, EqiForge achieves a geometric mean speedup of 1.32x and a maximum of 2.74x over the fastest available baseline per configuration. Its attention kernels outperform FlashAttention by up to 1.87x in decode and approach its performance in prefill. EqiForge also discovers new implementations that outperform torch.compile on various Transformer layers, including QK-normalized MLA (3.16x) and mHC (5.84x)."
W7214945532,10.48550/arxiv.2609.37852,10.48550/arxiv.2609.37852,2026,Delta-Matching: Closing the Final Gap of Native 8-bit Training for LLMs,Haozhan Tang,5,arXiv (Cornell University),0,"Reliable FP8 attention remains a barrier to fully native 8-bit large language model training. We derive how forward-backward inconsistencies produce stale delta and empirically show how it distorts training dynamics. Our stale-delta hybrid runs show a modest loss gap at 569M parameters but substantial loss increases and downstream degradation at 1.67B and 5.29B. QK normalization, NoPE (no positional encoding), and lower-learning-rate context extension mitigate or delay degradation without eliminating it. This pattern suggests accumulated optimization error that smaller models and short runs can conceal. We propose Delta-Matching, proving that it restores the softmax gradient's zero-row-sum invariant under the stated numerical assumptions. It enables native block-scaled FP8 in every forward and backward attention-core matmul without architectural changes, smaller global batches, or auxiliary forward outputs. Across tested architectures, scales, and training stages, Delta-Matching matches BF16/FP32 mixed-precision training loss and overall downstream performance. We will release our implementation, trained models, and data recipes."
W7196936801,,10.52202/085713-2546,2025,Learning in Compact Spaces with Approximately Normalized Transformer,Jörg K. H. Franke,6,neural information processing systems,0,"The successful training of deep neural networks requires addressing challenges such as overfitting, numerical instabilities leading to divergence, and increasing variance in the residual stream. A common solution is to apply regularization and normalization techniques that usually require tuning additional hyperparameters. An alternative is to force all parameters and representations to lie on a hypersphere. This removes the need for regularization and increases convergence speed, but comes with additional costs. In this work, we propose a more holistic, approximate normalization via simple scalar multiplications motivated by the tight concentration of the norms of high-dimensional random vectors. Additionally, instead of applying strict normalization for the parameters, we constrain their norms. These modifications remove the need for weight decay and learning rate warm-up as well, but do not increase the total number of normalization layers. Our experiments with transformer architectures show up to 40% faster convergence compared to GPT models with QK normalization, with only 3% additional runtime cost. When deriving scaling laws, we found that our method enables training with larger batch sizes while preserving the favorable scaling characteristics of classic GPT architectures."
W7223564422,,10.5281/zenodo.23541766,2026,"The Transformer Abides: What Attention Needs, and Why It Endures",Neil D. B. Bruce,1,Zenodo (CERN European Organization for Nuclear Research),0,"Nine years after its introduction, the transformer remains the default sequence model. The usual explanations are real: parallel matrix multiplication on accelerators, predictable returns to scale, a co-evolved ecosystem and exact recall from an uncompressed context. But they do not explain why most modifications barely move accuracy, which properties are essential, why the changes practice adopted took the forms they did, or where the design can still move. This paper is the entry point to a collection of papers that treat a token-mixing layer as an averaging operator rebuilt from its own input and iterated through depth. Such a layer faces six failure modes (collapse to consensus, contamination of content by routing, runaway feedback, the inability to abstain, dilution as context grows, and the inability to represent direction). Each has a protection, proved within a declared class, and a signature measured at small scale when that protection is removed; for gain control, which protects against runaway feedback, that signature is synthetic. We read softmax attention in a pre-normalized residual block as the one known point where all six protections are cheap at once under the prices of parallel hardware and gradient descent; this is an interpretation, not a measured optimum. Two mechanisms, measured at small scale, help explain why its neighbourhood is flat: trained networks compensate local deficits through spare capacity, and, in ten scored tests, making a protection explicit did not beat a block that already supplies it. The anchor, the residual or input path that protects against collapse to consensus, is the one protection whose loss compensation provably cannot repair: under strict mixing, where repeated averaging contracts token differences toward zero, no extra capacity of bounded gain (bounded sensitivity) brings back what is lost without it, and in small-model ablations nothing else in the block made up for it. This is a proved bound; a trained test that adds capacity is pending. Many changes adopted in the mixing layer and its block since 2017, among them pre-normalization, QK-norm, sinks and output gates, read as re-supplies of the same protections; the account is silent on others, such as changes to the MLP. The room that remains for the design to move is where a protection is patched (met only by a fix that works over a limited range, such as the trained length) or paid at a price that another design could avoid. It includes abstention and retrieval at long context, governed by a two-sided log-N law in the context length (with bounded scores, no fixed rule for abstaining keeps both as the context grows), and bounded memory, whose price for exact recall is a per-head capacity. It also includes decisions that sum over all competitors where reading the largest would do, and the writer of new features (in a transformer, the MLP and the output map), for which the collection has no theory yet of what a good writer computes. Along the way the collection establishes results that stand on their own: attention has an arrow that default statistics of a single operator cannot see; depth is a distribution over path lengths; converged adaptive averaging can amplify changes in its input and overturn correct predictions; and softmax obeys length laws that decisions on maxima escape while the competing scores stay below the decision's threshold. We state what is proved, what is measured, what is interpretation, and what is still pending. This is a working draft, not a final version. It is unpolished in places, incomplete in others, and keeps exploratory material a polished paper would leave out, because that material helps decide what comes next and shows the programme's big picture more fully. Please read it with that in mind: the work will be refined further, and later versions will appear under this record. It belongs to a collection released together on Zenodo: the entry paper The Transformer Abides, the monograph A Spectral Theory of Token Communication, and the papers Attention Sinks at Long Context, The Transformer as a Constrained Equilibrium and Beyond Transformers. This version is the text; the records, the code and the formal proofs follow in a later version."
W7214054473,10.48550/arxiv.2609.23314,10.48550/arxiv.2609.23314,2026,ValueDiff: Value-Geometric KV Cache Eviction for Sink-Suppressed LLMs,Junyoung Park,3,arXiv (Cornell University),0,"Modern LLMs with QK-normalization, gated attention, learned attention sinks, or logit softcapping exhibit weaker persistent attention sinks, on which existing KV cache eviction methods primarily rely. We observe that across these models, weaker sinks co-occur with greater value-vector dispersion relative to key-vector dispersion. Motivated by this value-side dispersion, we present ValueDiff, a value-geometric eviction that ranks tokens by the L2 deviation of their value vectors from the cache mean. The same score arises as the minimal-disturbance eviction under a max-entropy assumption about future attention. We evaluate under fixed cache budgets, with eviction at every block boundary during prefill and at every decoding step during generation. On RULER at a tight 2k token budget, ValueDiff retains 88-99% of dense across seven sink-suppressed models (best on 6 out of 7). On LongBench at the 4k budget, ValueDiff averages 92% retention across sink-suppressed models versus 83% for the strongest prior baseline. On MATH-500, ValueDiff is the strongest non-dense method on every sink-suppressed model tested at the 25% cache budget, outperforming prior methods by up to ~20 points on gated-attention models. Across all three benchmarks, value geometry emerges as the more reliable query-invariant eviction signal for sink-suppressed models."
W7165368044,,10.1007/978-3-032-38224-5_42,2026,QG-MIL: A Gated Transformer Aggregator for Domain-Agnostic Multiple Instance Learning in Medical Imaging,Luca Zedda,7,Lecture notes in computer science,0,
W7197353311,,,2025,Cost-Effective Development of Educational Chatbots: Training and Inference Optimization,Aibek Abilgazym,1,Nazarbayev University Repository (Nazarbayev University),0,"Large Language Models (LLMs) have revolutionized natural language processing tasks and show promise for interactive language learning applications. However, training these models and deploying them on resource-constrained devices (like smartphones or Raspberry Pi) is challenging due to their size and computational demands. This thesis presents an accelerated training pipeline for the GPT-2 124M model – the small- est GPT-2 – enabling it to be trained to competitive performance in under 3 minutes (using 0.8 billion tokens) on high-end hardware, a dramatic reduction compared to the original GPT-2 which required about 40 GB of text and multiple days of training. We achieve this speed-up through a combination of architectural optimizations (e.g. Rotary Positional Embeddings, ReLU² activations, QK-normalization, FlexAttention mechanism, and etc.), a custom Muon optimizer for faster convergence, and strate- gic hyperparameter tuning. The model was trained on the cleaned and deduplicated FineWeb dataset. The resulting model was then quantized to 8-bit, reducing memory by a factor of four with minimal loss in language modeling accuracy. We demonstrate the model’s effectiveness in an educational context, focusing on tasks like interac- tive conversation practice and grammar correction, and deployed on edge devices, including an 8GB Raspberry Pi. Benchmarks show that the optimized 124M model maintains language modelling performance (HellaSwag benchmark) while achieving real-time inference speeds within the limited memory ( 1 GB) e of these devices. This work highlights that with the right optimizations, even moderately sized LLMs can be trained and run off-line on personal devices for language learning, bridging the gap between large cloud-based models and accessible on-device AI tutors."
W7223920549,10.48550/arxiv.2610.34582,10.48550/arxiv.2610.34582,2026,HobbyLM: A Sparse Mixture-of-Experts Model Family on a Hobby Budget,Harish Santhanalakshmi Ganesan,3,arXiv (Cornell University),0,"We describe HobbyLM, a family of small sparse Mixture-of-Experts (MoE) models trained from scratch with a few hundred to a few thousand dollars of rented GPU time. The core is a decoder-only transformer with fine-grained experts, sigmoid routing with auxiliary-loss-free load balancing, grouped-query attention with per-head QK-normalization, and aMuon/AdamW optimizer split. Each of these choices was selected with a single-variable ablation suite at 130M parameters. On this architecture we train a 500M-parameter model (169M active) on 40B FineWeb tokens and post-train it into a chat model, a masked-diffusion language model, a function-calling and computer-use agent, and a single ""omni"" checkpoint that reads text, images, video, sound and speech through frozen encoders and small projectors. We run these models on a laptop CPU with a from-scratch Rust inference engine. Two of the authors then adopted the architecture without architectural changes to build HobbyLM-1B (1.04B total, 305M active parameters), trained on a 100B-token curated mixture with a high-quality anneal. HobbyLM-1B reaches a 47.5 average on seven zero-shot benchmarks, and its builders also release an instruction-tuned model and an on-device macOS computer-use adapter. Finally, we describe HobbyLM-Image, a 333M-parameter flow-matching diffusion transformer that generates 1024x1024 images in a 32x-compressed latent space, together with a dependency-light CPU implementation of the full text-to-image pipeline. We report what worked, what did not, and where the small-scale ceilings lie. We also state for each component which author designed or built it."
W7170035928,10.48550/arxiv.2607.18363,10.48550/arxiv.2607.18363,2026,A Controlled Study of Attention-Only Transformers,Henry Ndubuaku,8,arXiv (Cornell University),0,"Feed-forward networks hold two thirds of a transformer's non-embedding parameters, yet the architecture has not received a necessity test that controls parameters, compute, and depth at once. We pretrain attention-only decoder transformers (Simple Attention Networks, SANs) against standard transformers matched separately for parameter count, training FLOPs, and depth (2 to 48 layers), for up to 105B tokens at 6M to 87M parameters. Deleting feed-forward layers in place is costly: the standard transformer leads by 0.47 nats at matched depth and 0.26 nats at matched FLOPs. Reallocating the freed budget into attention depth closes the gap: at matched parameters the difference is 0.006 nats (0.27 percent of loss), reproducible to one part in ten thousand across seed pairs, shrinking across 5B, 30B, and 105B budgets, and holding near 0.02 nats across a 29x size range. Three measurements localize the remaining gap to parametric recall: attention-only models are better on context-grounded answers and worse where knowledge must come from weights. Weight spectra show why: routing matrices (Q/K) crystallize early, content matrices accumulate rank slowly, and removing feed-forward layers relocates this accumulation to the attention output projection. QK-normalization, not feed-forward layers or residual gating, keeps 48-layer attention-only stacks trainable. The deficit concentrates on low-context query prediction and localizes there entirely by the largest budget. A pre-registered test confirms the account: it predicts a 0.02 to 0.05 nat gap on knowledge-dense web text; a matched pair trained on fineweb-edu measures 0.040. Within the tested regime, attention does the rest."