Masters_Thesis/02-literature/search/results/snowball-fwd-zhai2023.csv

1	s2_id	arxiv	doi	year	title	first_author	n_authors	venue	citations	abstract
2	f89fc91cf6c7cede27bf44401819c504e5cbdca1	2610.31455		2026	MixPT: Point Transformer with Interleaved Window and Global Attention	Minkyun Seo	3		0	Point cloud transformers aim to capture detailed local geometry and relationships across a scene at a manageable computational cost. Two common approaches are window attention and global attention. They differ in how they enable scene-wide interaction, which refers to information exchange among tokens across a scene. Window attention reduces computational cost by restricting self-attention to groups of tokens called windows; in serialized designs, each window contains a fixed number of consecutive tokens from a spatially ordered sequence. Because each block only directly connects tokens within the same window, information exchange across windows relies on changes in window membership across blocks and hierarchical pooling. Global attention, in contrast, directly connects all tokens within a scene, but computing attention over all token pairs can be costly when the number of tokens is large. To balance computational cost and scene-wide interaction, we introduce MixPT, a point transformer that uses a mix of window and global attention blocks in an interleaved schedule, while retaining a single fixed serialization pattern for its window blocks. For scenes with a large number of tokens, MixPT applies global attention to spatially pooled tokens and returns residual updates to the original-resolution stream. Across indoor and outdoor benchmarks, MixPT achieves competitive accuracy with lower latency than representative window and global attention baselines. We further demonstrate the feasibility of scaling the attention stage in depth by repeating the interleaved pattern of window and global attention blocks.
3	860f8021d1a8cd3dec4a21981c669e8dab1a0449		10.1007/s13042-026-03333-9	2026	Graph-guided sparse attention variational autoencoder with causal-spectral fusion for unsupervised battery fault detection in new energy vehicles	An-Qin Zhang	3	International Journal of Machine Learning and Cybernetics	0
4	24f3d1102acc9f9882b039a0a4a97b5e8bdc633e	2610.22191		2026	Attention over noisy examples: generalization and memorization in the exponential Hopfield network	Francesco Alemanno	1		0	In Hebbian learning from examples, a Hopfield network retrieves archetypes seen only through noisy examples. We study it in the exponential Hopfield model: one softmax attention step with the unlabelled examples as keys. A signal-to-noise analysis gives the overlap with an unseen archetype in closed form: a majority-rule law in which an attention entropy replaces the dataset entropy. At high temperature the archetype is lost in the crosstalk; at low temperature the softmax freezes on the closest training example, as in a random energy model, and the layer memorizes. In between, in a window widening with the square root of the network size, the layer pools the examples of the correct class and retrieves the unseen archetype as well as their majority vote allows, at any fixed load. On structured datasets, such as MNIST and fashion-MNIST, the best accuracy lies beyond the freezing.
5	bcba462f8b88e55de5ab1f2d5cc1e9dd6b2c793f	2610.22330		2026	Designing Transformers for Continuous Diffusion Language Models	Wei-Jian Luo	5		0	Continuous diffusion language models (CDLMs) offer an alternative approach to parallel language generation; however, most existing studies directly use transformer architectures from either causal large language models (LLMs) or Diffusion Transformers (DiTs) for image generation, which were not optimized for CDLMs. In this paper, we conduct a systematic study on Transformer architectures for CDLMs and introduce Language Diffusion Transformer (LangDiT), combining feature and query-key normalization, wider noise conditioning, likelihood-shortcut gradient routing and calibration, and confidence-gated training self-conditioning. LangDiT significantly reduces generation perplexity in matched LangFlow comparisons, establishing a new state of the art in continuous language generation across different model formulations. Using LangFlow as a representative CDLM, LangDiT as a plug-and-play backbone leads to a new state-of-the-art model with a generation perplexity (GenPPL) of 21.72 using 1024 NFEs and 44.72 GenPPL using 128 NFEs on the unconditional generation of 1024-token sequences on the OpenWebText dataset. When adapted to Plaid-style CDLM formulations, LangDiT results in a new leading GenPPL of 11.33 using 1024 denoising NFEs, and 27.51 GenPPL using 128 denoising NFEs. These gains are achieved while retaining each model's underlying representation, training objective, and inference sampler, isolating transformer architecture as a fundamental source of improvement. Our study identifies architectural design as an important source of improvement in continuous diffusion language models.
6	6b4ef3210d16a284d304000fcd00daccbc945fed	2610.17588		2026	Taming Softmax at the Limits of Muon: A Unified Approach through Hilbert Geometry	Aditya Somasundaram	5		0	Muon controls weight updates through the RMS movement of a linear layer's output. This implicitly treats the linear output as a proxy for the full module, including the nonlinearity that follows. This proxy can be misleading for softmax, where the RMS shift in activations does not faithfully measure the change in the output distribution. We propose Hilbert distance as a natural metric for bounding changes in these output distributions. Hilbert distance bounds changes in cross-entropy loss and limits probability redistribution. For softmax, it takes the form of the variation seminorm of the preactivation perturbation, allowing probability constraints to be expressed as parameter constraints. Formulating the steepest descent problem with a Hilbert distance constraint brings two recently proposed methods under a unified mathematical framework. For the LM head, solving a row-wise approximation to the resulting diameter-constrained problem leads us to RowNorm \citep{pethick2025trainingdeeplearningmodels}. For scaled dot-product attention, the same principle yields the first-order query--key steepest descent problem underlying Compositional Muon \citep{keigwin2026compositional}. These connections motivate using Hilbert distance as a guiding principle for optimizer design in softmax modules.
7	7cee73c17cc345818796a6a1325020c0460cb738	2610.13518		2026	Asymptotic Analysis of Discrete-Time Self-Attention Dynamics in Transformers	Claudio Altafini	1		0	In machine learning, a self-attention dynamics describes the attention mechanism of transformers, and is obtained treating the layer index of a transformer as time index. Even though naturally discrete-time, this dynamical system is most often studied in continuous-time. Scope of this paper is to provide a thorough investigation of the discrete-time version of the self-attention dynamics: a complete classification of its equilibria is given, and their stability properties are studied, in both the single-head and the multihead case. We show that in both cases the main attractor of the dynamics is a consensus equilibrium aligned with the principal eigenvector of the value matrix (in the single-head case, a corresponding matrix of weights in the multihead case), even though other locally asymptotically stable bipartite equilibria can be show to exist at least in the single-head situation. The behavior is confirmed by looking at the learned weight matrices of a real transformer.
8	5854ef7937434b160f98cde453d1c9b02a502a75	2610.11205		2026	Dissecting Representation Structure in Vision Transformers: A Rigorous Architectural Study	Kim Nguyen	2		0	Representation structure is crucial for understanding Vision Transformer (ViT) architectures and their generalization behavior. However, prior studies neither isolate nor analyze module-level features nor investigate how their interactions contribute to performance estimation. In this work, we conduct the first rigorous analysis of feature information across diverse architectural scales, empirically uncover the relationship between ViT representation and generalization behavior, and leverage these insights to guide efficient ViT design. Our contributions are fivefold: Across diverse architectural scales, 1) We identify feature collapse at initialization, which leads to redundancy, and propose a reduction scheme to mitigate this issue. 2) We quantify feature information using entropy and the minimum eigenvalue, demonstrating that these metrics serve as reliable indicators for generalization prediction. 3) We show that feature in the token space provides a more faithful representation than those in embedding space. 4) We discover an unexpected finding: features produced by linear submodules within ViT layers are critical for the prediction of generalization performance. 5) Our proposed proxy improves the correlation ranking by 18-48% over prior baselines and can effectively identify ViT architectures that achieve higher accuracy at lower or comparable computational cost.
9	73be128360856cf860401daa323151e09e95fae2	2610.09958		2026	Force without transmission: a depth-induced rank collapse that no loss on the representation reopens	Martin Hofmann	2		0	Training can drive a transformer into a rank collapse: all token representations point in one direction, and learning stops. In a related collapse of attention, a loss term with a bounded corrective force repairs the network during the run. We ask whether such a term repairs rank collapse. We collapse small transformers by weakening their skip connection and treat copies of the collapsed network. No added loss term repaired the collapse, although the stronger kind pushed with about a tenth of the task gradient. The reason was the path, not the strength. The task gradient no longer reached the query and key weights, which decide where attention looks, and the added term's gradient faded before the blocks where the collapse forms. Restoring the skip connection, which changes no weight, reopened this path at once. The rank then recovered, but only far above the scale of collapse. After a burst of high learning rate the path stayed open and the rank recovered untreated. Registered predictions from the path ranked recovery times but did not transfer to this cause. In every case the loss stayed above that of a healthy network after the rank recovered. Whether a collapsed network can be repaired depends on whether the gradient still reaches the weights that must change, not on how strongly a loss term pushes.
10	97513573486d7ca59d109dfd4a635373fc077203	2610.06012		2026	Spectral Geometry of Attention: From Information Routing to Uncertainty	Giulio Viganò	3		0	In this work, we study transformer attention through the lens of spectral geometry and operator theory. We view each attention head as a functional map between Hilbert spaces of functions on the token sequence and derive a Token Difference Operator, whose spectral structure controls how token-space information is routed to the output. We show that standard Euclidean spectra are structurally biased by sinks, conflating mass concentration with genuine routing capacity. By recasting token space in the intrinsic probability geometry induced by attention, the token difference spectrum disentangles sink effects from routing capacity and provides a spectral description of the dimensionality of the head output. This yields a unified framework for analyzing attention maps, explaining sinks, routing collapse, and output dimensionality within a single operator-theoretic framework. In practice, by grounding attention heuristics in spectral geometry, we develop a novel attention-based uncertainty estimator that complements probability-based scores, with the largest gains on long-context inputs.
11	fc7df63b37f90ccd867d5cfcdf60ddbdb866f18f	2610.04424		2026	On the Trade-off Between Information Loss and Generalization in Sparse Attention	Zhong-Qi Fan	2		0	To mitigate the quadratic complexity bottleneck of the Transformer, sparse attention has emerged as a pivotal technology. Despite the extensive empirical success of sparse Transformers, the theoretical understanding of sparse attention remains fragmented. In particular, two fundamental questions remain unclear: (1) How does sparsification affect the information fidelity of attention mechanisms? (2) How does this information loss interact with the generalization behavior of the model? To bridge this gap, this paper proposes a systematic analysis of the Jensen-Shannon (JS) divergence and of the generalization gap of sparse attention mechanisms. Specifically, we first characterize the approximation error via the JS divergence. Through an order-statistics-based concentration analysis of the truncation mass alpha --- where the attention scores are assumed to be independent and identically distributed sub-Gaussian random variables with parameter sigma --- the JS divergence between the full attention distribution and the sparse attention distribution is shown to admit the closed form log 2 + ((1 - alpha)/2) log(1 - alpha) - ((2 - alpha)/2) log(2 - alpha). Subsequently, we derive a generalization bound through Rademacher complexity, quantified by O(gamma * sqrt(M/n) * (sqrt(log(3eL/M)) + sqrt(pi)/2)). Furthermore, building on a mutual-information-based generalization bound together with an entropy and covering-number analysis of the sparse hypothesis class, we obtain the sparsity-dependent generalization bound O(sqrt((M/(2n)) * (log(eL/M) + log(1 + 2/epsilon)))). Our analysis shows that sparsity reduces the complexity of the considered hypothesis class while introducing approximation error that can be quantified by the JS divergence. These findings provide a theoretical characterization of the trade-off between information fidelity and generalization in sparse Transformer architectures.
12	eba70174b120f76d0dc40c7c9381df292e22e0cc	2609.37976		2026	$S^3$: Spectral Null-Space Swap Makes Reasoning Models Efficient	Hong-Bo Ma	5		1	LLMs trained with Chain-of-thought excel in reasoning capability, but often come with excessive token cost. We find that the core of reasoning capacity lies in the Thinking model's weight component within the null space of a projection defined by the corresponding Non-thinking model's dominant singular directions, and removing the subspace component can largely improve reasoning efficiency without hurting the accuracy gained during thinking-mode post-training. Unlike existing efforts that mostly operate within the dominant subspace, we are the first to unveil the critical role of the null space and harness it for model optimization. Motivated by this finding, we propose Spectral Null-Space Swap ($S^3$), a training-free composition of paired Non-thinking and Thinking checkpoints. Our method keeps the Non-thinking model inside its own dominant subspace and takes the Thinking checkpoint outside it, improving reasoning efficiency while maintaining accuracy. We extensively evaluate $S^3$ on 2B-30B dense and mixture-of-experts (MoE) architectures spanning 28 evaluation environments across mathematical, multimodal, and audio reasoning domains. $S^3$ establishes new empirical Pareto Frontiers among training-free model composition strategies: across all settings, it reduces inference token overhead by an average of 27.4% compared to full Thinking models while simultaneously improving overall task accuracy by 1.0 percentage point (e.g., yielding +8.3% accuracy on HMMT25 alongside a 33.0% token speedup). We further use attention entropy for explanation and find that the retained component produces more concentrated attention, and we use a simplified analytical model about optimization to demonstrate why null-space can effectively reduce attention entropy, thereby improving the efficiency of reasoning.
13	b166abba59f18102e843c2326af480bfbaffb7fb	2609.34273		2026	Query Expansion and Key Specialization in Transformer Attention Geometry	Vidit Gupta	5		0	The projection of queries and keys are central to the attention mechanism in Transformer architectures. While they are mathematically symmetric, they play different roles in attention mechanisms. The question of whether there is an effect from their functional distinction on their geometric development in training remains unanswered. We investigate the problem through the training of small GPT-like Transformers on character-level WikiText-103 for three different depths (4, 6, and 8 layers), three types of initialization for queries and keys, and four random seeds, resulting in 36 runs and 54 trajectories of average layers across seeds. We track the effective dimensionality of those layers using participation ratios and discover that effective dimension of queries expand while keys shrink, and that $PR_Q - PR_K$ is positive in all trajectories studied. In connection to attention, the shrinking of keys leads to a narrower spectrum of $QK^\top$ and more peaked attention weights. In order to determine if this connection is causal or coincidental, we directly control the spectrum of keys during training across five seeds: restricting it to make it shrink sharpens the attention with high directional confidence, while keeping it constant to the level of initial dispersion makes attention softer. Additional token-level checkpoint analyses show that the monotonic paired-contrast trend is not universal across pretrained families, but survives as an early-training regime that later decays over a full pretraining run, and the link between interaction-rank geometry and attention entropy remains visible in several models.
14	7be54510c483ee2f1389f41f7427ca5579e1ff11	2609.34272		2026	Broken Symmetry in BF16 Attention: Why FlashAttention Gradients Blow Up Late in Training	Jun-Lin Chen	12		0	BF16 is now standard in large-scale pretraining, including in fused attention kernels such as FlashAttention, and these kernels are widely trusted. When we used FlashAttention-3 to pretrain a 450M-parameter transformer on 50B tokens, however, we ran into a problem: training was healthy for 25B tokens, then the gradient norm grew a thousandfold and the loss ended 0.2 nats above FP32 attention, without a single NaN. Recomputing the attention backward of just two layers in FP32 removes almost all of the excess gradient. Part of the cause is known: a fused multiply-add in the forward softmax, so far treated as an extreme-input NaN case and never fixed in FlashAttention-3. Repairing it stops the blow-up, but the query gradient is still wrong by more than its own size, and training still drives attention logits to thousands of times their size under accurate gradients. The remaining error comes from a broken conservation law. The softmax score gradient sums to zero along every row, which makes the query gradient blind to where the keys sit as a group; rounding it to BF16 leaves a small nonzero sum that leaks the mean key into the gradient, and the leak grows exactly as late training makes keys large and attention sharp. We introduce GProj (gauge projection), which restores the zero sum after the cast with two rank-one corrections per row. It cuts the remaining median query/key gradient errors from 219%/13% to 0.34%/0.37%, on par with FP32 attention, for 4.7% more time per training step. In matched from-scratch runs it trains to the same loss as FP32 attention, while FlashAttention-3 and key smoothing both destabilize.
15	5374dcf4fe294464395cbca210a4a453d7beac20	2609.30592		2026	QSV: Quat-Sphere-Vision for Coupled Quaternion Attention on Spherical Lattices	Nicholas Foley	5		0	In standard attention, three separately learned projections decide how strongly a token attends to each neighbor ($W_Q$, $W_K$) and how the attended features are transformed before aggregation ($W_V$). We study Quat-Sphere-Vision (QSV), a sparse spherical vision model that replaces this projection triple with a single learned unit quaternion per token: the relative quaternion $r_{ij} = q_i^{*} \otimes q_j$ supplies both the attention logit $\operatorname{Re}(r_{ij})$ and a sandwich-product feature transport $x \mapsto r_{ij} \otimes x \otimes r_{ij}^{*}$, with messages passed over sparse kNN graphs on concentric Fibonacci spheres. Ablations that change only the targeted component show the two roles to be asymmetric. Removing the transport reduces test accuracy by about four percentage points on CIFAR-10 and CIFAR-100 (single runs per CIFAR-100 variant), while replacing the learned attention weights with uniform averaging leaves it essentially unchanged. Parameter-matched controls then remove the geometry itself: standard attention on the same graph exceeds QSV (mean $87.3\%$ vs. $85.9\%$), and the same model on a flat 2D lattice reaches $91.1\%$, within $2.1$ points of a ResNet-20 trained under the same pipeline (single run). In the coupled kernel, nearly all of the learned pairwise computation resides in the transport channel.
16	43d380b4351c6e5d72d454973a888ca95a6f8efb	2609.20086		2026	SETTer: Sparse-Encoder Transformer for Long-term Multivariate Time Series Forecasting	Abraham Ezema	4		0	Long-term multivariate time series plays a significant role in many application areas such as power systems, trading, etc. However, their accurate prediction is quite difficult for conventional forecasting methods as they often exhibit high dimensionality and complex relationships. Recent works show that transformer-based approaches are quite effective for long-term forecasting thanks to their attention mechanism. However, in the presence of complex high-dimensional inputs, they show evidence of oversmoothing, limited capacity, and opacity. To this end, this paper introduces SETTer, a transformer-based model that addresses these challenges by incorporating novel techniques for decoupled self-attention and hybrid masking. The proposed techniques enable SETTer to effectively capture the dominant short- and long-term patterns across the temporal and channel dimensions. In addition, we enrich the model layers with simple explainable structures that indicate the discriminative pattern of SETTer. We show that with a single-layer transformer architecture, SETTer can effectively model long-term dependencies in the presence of varying data complexities. Extensive experiments on real-word benchmark datasets for long-term multivariate time series forecasting demonstrate that SETTer outperforms state-of-the-art models in 88% of the scenarios.
17	45168d8a6ccbcaaeb4ead00f4c35b4a4f7bbc2cc	2609.19363		2026	Directions That Don't Drift: Stiefel Manifold Routing for Transformer Attention	Rubén Darío Guerrero	1		0	The query and key projections $\WQ,\WK$ in attention are almost always trained by Euclidean optimizers with no geometric constraint. We constrain them to the Stiefel manifold and optimize with a Riemannian Adam carrying one scalar second moment per frame---the form of \citet{becigneul2019}, here extended to the compact, non-Hadamard $\St(d,r)$ with a tangent projector, step-norm cap, and polar retraction. Four propositions prove steepest descent in the embedded metric, gradient-scale independence, well-conditioning, and exact $\mathrm{O}(d)$-equivariance. A fifth records that weight decay has \emph{identically zero} Riemannian gradient on $\St(d,r)$ ($W{=}WI_r$ lies in the normal space), so decay cannot act on the constrained frames. On a CIFAR-10 patch benchmark at $n{=}10\mathrm{k}$ this rule gains $\mathbf{+6.79}$\,pp over AdamW across 12 paired starts ($t{=}38.33$, $12/12$); earlier fixed-step Riemannian SGD gains $+1.97$\,pp, of which $+1.69$\,pp comes from frozen orthonormal initialization alone. The corrected Adam's lead grows with data: $+1.9$\,pp at $n{=}1\mathrm{k}$ to $+6.7$\,pp at $n{=}50\mathrm{k}$. A 12-seed ablation credits all gain to the scale-free step ($+4.63$\,pp, $12/12$), nothing to the projector or equivariance; a targeted $\varepsilon$-sweep causally confirms the mechanism ($-2.6$\,pp at $\varepsilon{=}0.1$, $p{<}0.001$). Two five-seed grokking studies confirm the constrained arm does not grok better than the baseline ($p{=}0.019$, A2 wins): the weight-decay exemption has no grokking consequence. A single-seed pilot exploiting this localization achieves the first stable grokking under slingshot conditions---Stiefel + targeted circuit regularization keeps routing-frame isometry error $10^6\times$ lower than the unconstrained ablation through every collapse.
18	38624d2075689a3151af1341ad96a6762091ea9b	2609.00746		2026	Text Capability Loss in Vision-Language Adaptation: An Attention-Sink Diagnosis	Minsik Choi	3		0	Fine-tuning a pretrained LLM into a vision-language model (VLM) can erode the backbone's text capability, with the damage concentrated on tasks that require following exact output rules, such as instruction following, chain-of-thought reasoning graded on a strictly parsed final answer, and similar evaluations with strict graders. We trace this gap to attention-sink corruption: VL fine-tuning perturbs the early sink position that anchors a large fraction of attention probability, and how well the base LLM preserves its sink tracks how much of the affected capability survives adaptation. Building on this view, we introduce Sink Strength, a single scalar computed on the base LLM in a few seconds on a single GPU that predicts post-VL degradation without any VL training. It consistently tracks relative degradation across the six VLM-LLM pairs and multiple format-sensitive tasks. Complementing this diagnostic, we find that post-pretraining QK-RMSNorm injection fails to reproduce the protection of native QK-RMSNorm, while several off-the-shelf weight-merging settings fail to recover the lost capability after VL training. These negative results underscore the value of screening backbones with Sink Strength before VL training and narrow the intervention space toward head-selective training-time protection.
19	36094f6b16332f9afcf0329fa605ea5fad5f0a15		10.1109/TBIOM.2026.3671674	2026	Revocable Biometric System Using Multiple Identical Deep Neural Matcher Instances	Aman Bhatta	4	IEEE Transactions on Biometrics Behavior and Identity Science	2	Biometric authentication systems face a critical security challenge: once a biometric template is compromised, it cannot be replaced. Revocable biometrics address this by enabling template cancellation and reissuance without degrading recognition accuracy. We systematically evaluate a revocable framework in which multiple independently trained instances of the same end-to-end model yield revocable templates. From a shared backbone architecture, training dataset, and identical hyperparameter configurations, we derive distinct face matchers that exhibit near-identical genuine and impostor score distributions. Notably, cross-model similarity scores (genuine scores) for the same individual are lower than same-model scores for different individuals (impostor scores), reducing the value of stolen templates to sub-impostor levels in cross-model scenarios. We assess ResNet-100, MobileFaceNet, and ViT backbones under ArcFace and AdaFace losses, verifying robustness across architectures and loss functions. t-SNE visualizations of embeddings from independently trained instances with the same backbone and loss show that features realize different coordinate systems on a shared face-identity manifold, achieving comparable validity and accuracy rather than representing trivial permutations or transforms. However, an adversary with paired feature vectors from two models for the same enrolled images can still approximate the inter-model transformation. To mitigate this, we propose a model-mixture deployment that randomly and uniformly partitions subjects across models, with subsequent updates following the same randomized partitioning to obstruct direct linear mappings while maintaining stable operation of the revocable biometric system.
20	33b03facc184393c55f5656d43a5bfdeaedace56	2608.21541		2026	Beyond Sparse Weights: When Is Attention Compressible?	Chi-Wun Yang	2		1	KV-cache compression is often justified by attention maps with a few large weights. This is incomplete: large weights may not contain most of the mass, omitted values can cancel, and preserving the attention output may not preserve the task. We separate these questions. Global score gaps -- not threshold counts -- determine how many tokens are needed to retain a target mass. For a realized row, the weighted sum of omitted values is the exact missing statistic. A controlled retrieval--aggregation model explains when truncation helps and when it hurts. These results motivate CertKV, a training-free compressor that reserves one tail-summary slot per head and allocates the rest by value dispersion. Under matched budgets, CertKV is top-two in seven of nine LongBench-v2 settings, remains in the leading compressed tier on 128K RULER, and realizes a ten-fold cache budget in a packed Llama prototype. Compressibility depends on the mass, values, future queries, and task -- not on a sparse-looking map alone.
21	47590b4d453fa4ec8f927e1b9eeeeb9746a00684	2608.18399		2026	What Does Attention Transfer Transfer? Attention Structure and Robustness in Vision Transformers	Jesse Ponnock	1		0	Vision transformers (ViTs) trained to copy a pretrained teacher's attention maps recover most of fine-tuning's in-distribution accuracy yet fall measurably short of it under distribution shift, as recent work has shown. What the copy delivers has never been measured directly in the attention structure and tied to robustness. We build that instrumentation for ViT-S students of a self-supervised teacher on ImageNet-100, and report three findings that triangulate one conclusion. First, the transfer is essentially perfect and permanently so: the distilled student's attention ends up roughly two orders of magnitude closer to the teacher's than fine-tuning does, and does not drift with additional training. Second, the gap is real at 14$\times$ fewer parameters and 10$\times$ less data than previously studied, but it has a time axis. It tracks training maturity, and completing the schedules that the stopping rule interrupted closes it below our pre-registered threshold in two of three seeds, with comparisons at equal accuracy giving the same result. The endpoint gap at this scale is substantially a training-maturity artifact: robustness matures later than accuracy, and stopping rules tuned to accuracy undersample it. Third, forcing cross-row redundancy down by half the structural separation between the distilled and fine-tuned conditions produces no detectable robustness response under two registered ways of matching accuracy. Verified transfer, a gap that closes while the structure never moves, and a null under direct intervention are together consistent with the deficit residing in features, not in the visible attention structure. This is elimination plus intervention, and its scope is the regime we measured. In this regime, attention overlays show where a model looks, not what it knows.
22	9e7c2160f9771f04b02869f23e8031d478ed5b48		10.3390/bdcc10080272	2026	Layerwise Conditioned Backpropagation: A Curvature-Aware Reparameterization of the Backward Pass with Convergence Guarantees	Maikel Leon	1	Big Data and Cognitive Computing	0	Backpropagation is less a single algorithm than a pipeline of choices: how the error signal is propagated, how the weight gradient is assembled, and how the update is applied. This paper revisits three consecutive steps and proposes small, mathematically transparent modifications that improve gradient scaling and conditioning without changing the represented function class. The resulting method, Conditioned Backpropagation(CBP), combines (i) a layerwise gradient-norm equalization that counters the geometric depth dependence of the backpropagated error; (ii) an activation-centering reparameterization that removes the dominant rank-one mean term from the per-layer curvature; and (iii) a damped diagonal preconditioner that is positive-definite by construction. The composite operator is a bounded positive-definite preconditioner, so the method inherits standard nonconvex, Polyak–Łojasiewicz, and stochastic convergence guarantees at the per-step cost of ordinary backpropagation. No prior method composes these three repairs into one operator with a joint boundedness and positive-definiteness guarantee. Two further results, both new, concern equalization. On a block-structured strongly convex model, and for the curvature-equalizing target that the implemented gradient-energy equalizer approximates up to a quantified heterogeneity factor, equalization makes the convergence rate depth-uniform; the bounded-clip version that is actually run stays depth-uniform up to a clip-determined depth and retains a constant-factor improvement beyond it. Controlled experiments, run over ten or more seeds with paired significance tests, confirm the mechanisms: Equalization compresses an order-of-magnitude per-layer gradient disparity, centering cuts the top curvature eigenvalue about threefold and yields the lowest training loss, and the configurations combining centering with the damped preconditioner, including the full method, converge fastest. The effects persist on MNIST and on CIFAR-10 with a small residual convolutional network, at a measured per-iteration overhead below about twice that of Adam. Generalization is comparable across methods, and no end-to-end depth-scaling advantage is claimed, keeping the contribution focused on optimization geometry.
23	527ca888f2f1eb583acd753a2c595ecce0595f04	2608.14712		2026	Which Question Is Your Attention Metric Answering? Attention Rows as Compositional Data	Marios Papamichalis	2		0	Each row of a transformer's attention matrix is a probability distribution over tokens, and in trained models most of that probability lands on a single \emph{sink} token, usually the first. Standard tools for comparing attention rows (cosine similarity, Jensen--Shannon divergence, Shannon entropy) therefore hinge on a choice papers rarely report: keep the sink, or drop it and renormalize. This choice can reverse conclusions. On ten pretrained models from five families, 17--47% of verdicts about which of two heads is more similar flip with the convention, and the most prominent structure in a standard BERT head-clustering pipeline is an artifact of it. The reason is that one-number summaries mix two questions: how much attention the sink takes, and how the rest is divided among the content tokens. Treating rows as compositional data separates them exactly: the Aitchison distance splits orthogonally into a sink term and a content term, entropy splits by an exact identity, and the content distance is characterized by invariances the transformer itself possesses. The separation matters in practice: most measured entropy collapse during training is the sink growing, not attention sharpening (30% of the drop at 70M parameters, 95% at 1B, 79% at 1.4B), and pruning heads with the wrong channel can inflate perplexity more than a hundredfold. We map where each convention is safe, test a frozen out-of-sample predictor (one confirmation, one abstention, one failure), and release code regenerating every number.
24	3116e912416adb43a345cc4d6bae036fce0f6bf1	2608.09417		2026	Why Post-Norm Transformers Collapse: Attention Amplification and Gradient Repair Failure	Xing-Jian Wang	4		0	Deep decoder-only Transformers often replace the original Post-Norm architecture with Pre-Norm variants because Post-Norm training is highly sensitive to warmup and learning rate under conventional initialization schemes. Although prior work has identified rank collapse and gradient vanishing as related symptoms, it remains poorly understood how causal attention creates high-similarity representations and why training dynamics fail to repair them. We give a two-stage analysis of Post-Norm rank collapse using token similarity as a scalar state variable. First, at initialization, causal attention acts approximately as a prefix-averaging operator that increases token similarity across depth, while the SwiGLU branch contributes only a smaller damping effect. Second, once training enters a high-similarity regime, growth of pre-normalization residual norms makes the RMSNorm backward factor contractive; under mild conditions, gradients to earlier layers decay geometrically. As a complementary result, we characterize the properties of a collapsed network: its best predictor is frequency distribution with relatively high loss floor, and gradients in collapsed layers vanish at frequency distribution. Experiments on 48-layer decoder-only Transformers trained on C4 dataset match the predicted initialization-time similarity growth and collapse-time gradient contraction, and show that collapsed runs stay near the predicted frequency loss. Together, these results distinguish the forward similarity amplification and backward repair incapacity in Post-Norm collapse, while also characterizing the behavior of collapsed networks.
25	54d8ac88d73529a89724ba705b1e3a1a3831b8fc	2608.08922		2026	Clustered Attractor Manifolds and Dynamical Condensation in Self-Attention	Qu-Cheng Gao	3		2	Transformer layers generate state-dependent interaction networks: token representations determine the attention matrix, which in turn updates the representations. We study this feedback in a minimal normalized self-attention dynamics and identify the overlap gap as the central quantity governing its attractor structure in the thermodynamic limit. When tokens form internally aligned clusters and their similarity to members of the same cluster exceeds that to every other cluster by a nonvanishing amount, inter-cluster attention is exponentially suppressed as the dimension increases. This mechanism produces a high-dimensional manifold of clustered fixed points, ranging from a few macroscopic clusters to extensive microscopic fragmentation, and also controls their stability against perturbations. Starting from an unstructured Gaussian state, we find that clustered states nucleate from the diffuse background only above a finite threshold in attention sharpness, giving rise to a dynamical attention-condensation transition.
26	044f1246854dfc49d396e88d4d823bc8df2d99f3		10.3390/signals7040076	2026	Representation and Geometric Collapse in Spatiotemporal EEG Classifiers: A Mathematical Diagnostic Framework	Ahmed El Badaoui	5	Signals	0	Spatiotemporal deep learning models, like Graph Neural Networks (GNNs), Transformers, and selective State Space Models (Mamba), have achieved impressive performance in electroencephalogram (EEG) decoding and affective computing. However, their generalization performance often degrades severely under subject-independent Leave-One-Subject-Out (LOSO) cross-validation protocols. This generalization drop is often attributed to generic domain shifts in standard Brain–Computer Interface (BCI) literature, and is tackled by parameter-heavy adaptations. In contrast, this paper proposes a unified mathematical diagnostic framework to audit and measure the underlying representation and geometric collapse in spatiotemporal brain–computer interfaces. More concretely, we formalize: (1) Topological over-smoothing under volume conduction through Graph Dirichlet Energy bounds indicating GCNs as low-pass filters that smooth localized electrode variations; (2) representation collapse through the Normalized Rank Uniformity Index (NRUI) based on the Shannon Entropy of latent covariance eigenvalues, that distinguishes between dimensional and semantic collapse; and (3) geometric manifold distortions under subject domain shifts on the Symmetric Positive Definite (SPD) Riemannian manifold under the Affine-Invariant Riemannian Metric (AIRM) projection. Auditing these diagnostic metrics on canonical models across DEAP, DREAMER, and SEED, we demonstrate why standard spatiotemporal architectures suffer from performance collapse in cross-subject configurations. We provide BCI engineers with a tangible mathematical blueprint to design robust, collapse-resistant decoders.
27	20bb621e677e2dd56997172bf80363020157f775		10.1007/s44163-026-01839-3	2026	Layer-wise symbolic attention instability as a diagnostic signal for hallucination in large language models	Naveen Lamba	3	Discover Artificial Intelligence	0	Large Language Models (LLMs) tend to hallucinate when processing symbolically complex linguistic structures. Existing literature evaluates hallucination either through their mechanistic interpretability or at the behavioral output level, but hardly links the symbolic triggers to their layer-wise representational causes. This paper introduces a unified symbolic, behavioral, and mechanistic framework that connects symbolic triggers with internal failure dynamics in transformer architectures. The study evaluates five open-weight LLMs across QA, MCQ, and Odd-One-Out formats on the HaluEval and TruthfulQA datasets, focusing on negation, exceptions, modifiers, numbers, and named entity cues. The results show that hallucination rates remain high across all model scales, with all symbolic categories exhibiting high hallucination rates, and exceptions and numbers often showing comparable or higher values across models. Constrained task formats reduce surface errors but preserve failure patterns, indicating representational instability rather than purely decoding artifacts. Layer-wise analysis shows peak symbolic attention variance in early transformer layers (2–4), after which these patterns persist across deep layers. The consistency of this behavior across architectures suggests that hallucination is strongly associated with weakness in symbolic encoding. The framework provides an interpretable basis for diagnosing and stabilizing symbolic reasoning in LLMs.
28	9653176d4dafd012f0b04775889b80cc7b8c6b91	2608.02091		2026	One QK Channel, Many Sources: Tracing Low-Precision Attention Collapse	Shu-Xiao Xie	6		0	A bfloat16 transformer can train normally, then collapse abruptly. Prior work links collapse to structured attention errors and shows QK normalization disrupts their compounding. Distinct low-precision errors trigger the same collapse, leaving unclear whether each needs a fix at its source or one shared route can be blocked instead. We isolated the fault behind a reproduced GPT-2-class collapse to the streaming-softmax accumulator, where an fp32 streaming core repairs it, and turned it into an assay for moving a controlled error across sources. Using it, we found that errors placed outside attention still drove the same QK spectral runaway, and that correcting only QK kept training stable while the fault stayed active. This is a source-channel dissociation: fault source is not failure channel. It held across tested architectures and scales, and reproduced on a second GPU architecture. As a causal probe, projecting each update off the current QK weights'leading three singular directions held the query projection's largest singular value to 11.1, whereas removing equal energy elsewhere left it at 237: the QK channel causally drives the early runaway. What lets the injected error in is temporal sign-coherence, its per-head sign persisting across steps, not aggregate deviation; once inside, the runaway shows as attention-logit saturation. QK-Guard, a dormant controller, tests this by switching on parameter-free QK normalization at the first monitored threshold crossing. On the runs designated for this test, the QK-local action prevented the failure of each matched or same-configuration unguarded run; on plain GPT-2, all 12 final train and validation losses were within 0.03 nat of same-configuration always-on QK-norm, and both methods ran 60k steps without collapse. Intervention at the QK locus therefore suffices in place of a fix at each source.
29	92b13ade43c96cd122902e5c25d6b4eaf3ea1bf1		10.1016/j.knosys.2026.116370	2026	PARCNet: Phase-aware residual correction network for efficient multivariate time series forecasting	Zi-Qiong Li	5	Knowledge-Based Systems	0
30	04224fe0abfc0121f0e174078641c8861e5a5a83		10.1016/j.eswa.2026.133986	2026	CQMTformer: Channel query enhanced Mamba-transformer fusion architecture for multivariate time series forecasting	Junyu Jin	4	Expert systems with applications	0
31	9677f1da98aa894c7ffc1dda70be29f59eeafaa1	2607.27805		2026	Chaos in reason: How chain-of-thought LLMs can look for an answer	Gregorio Jaca	3		2	Large Language Models (LLMs) have achieved remarkable performance across a wide range of tasks, yet their internal dynamics remain poorly understood. In this work, we apply the tools of nonlinear dynamics and chaos theory to LLMs. By analyzing both text and hidden state trajectories, we demonstrate that LLMs exhibit hallmark signatures of chaos, including strong sensitivity to initial conditions, manifested as intermittent, jump-like divergence of nearby trajectories combined with bounded evolution, with consistent results across different distance metrics. An exact Jacobian analysis of the Transformer's sub-blocks shows that self-attention and the feed-forward network expand and propagate perturbations, while normalization and residual connections counteract this expansion and promote stability. Recurrence plots show structural similarities between LLMs and canonical chaotic systems such as the Lorenz attractor, while dimension analysis reveals fractal structures in the hidden state space, particularly pronounced in the last layers. We propose that the nonlinear coupling induced by attention mechanisms plays a key role in driving this chaotic behavior.
32	e3121c704b2f1c0bdf7b67a819387855bccb8124	2607.01420	10.48550/arXiv.2607.01420	2026	MultAttnAttrib: Training-Free Multimodal Attribution in Long Document Question Answering	D. Tran	10	arXiv.org	2	As grounded QA systems are increasingly deployed in AI assistants, accurately attributing generated answers to evidence is critical for user trust and model safety. While unimodal attributions have been explored in depth, the multimodal setting remains relatively under-researched. As a result, we introduce MultAttnAttrib, a training-free attribution-generation method that leverages a model's prefill pass, selected attention heads, and calibrated thresholds to locate source evidence within a document. To establish baseline results for the method, we introduce MultAttrEval, a complementary benchmark dataset annotated with fine-grained, ground-truth attributions for answer components grounded in multimodal source documents. To our knowledge, this is the first evaluation dataset designed specifically for multimodal attribution in long-form documents. Experimental results show that MultAttnAttrib consistently outperforms a variety of attribution-generation methods, including several strong prompting-based approaches and matches the latest frontier models such as GPT 5.4. Our method not only substantially improves attribution accuracy for both unimodal and multimodal attribution types, but also produces attributions at up to one-seventh of the direct inference latency compared to prompting on the same base model.
33	80ef27eadc8c5276feb7915d11d1bc98c0cdc1d0		10.1007/s11119-026-10364-3	2026	Constructing a Cross-Instrument Transfer Learning Method for Chlorophyll a Fluorescence OJIP Transient Using OJIP-SNV and CGAN: A Case Study for Cotton Salt Stress Diagnosis	Jia-Yi Li	9	Precision Agriculture	0
34	5e1e83d96c3b5b43c0feb8a0e9e5a78bef76ea3c	2606.28116		2026	Mechanism-Aligned Observability for Training Instability of Large Language Models	Rui-Xuan Huang	8		1	Frontier large language model training consumes massive accelerator fleets over long wall-clock times, making stability failures extremely costly. However, global symptoms such as loss divergences often emerge only after the underlying fault has already affected the training dynamics, further increasing the sunk compute. In which internal representation can a training fault become persistently and interpretably observable before it appears in global training metrics? We propose a mechanism-aligned principle for training observability: identify the internal state into which a fault is persistently written, derive the structure induced by its mechanism, and construct monitors in representations aligned with the computation of the affected module. For FlashAttention, we show that biased backward errors accumulate into a persistent low-dimensional structure in weight updates, producing spectral concentration. For MoE routers, we characterize routing dynamics through per-token routing behavior rather than parameter similarity, which may change without altering the routing function. Experiments show that the two monitors respond selectively to changes in their corresponding modules, before degradation becomes visible in global training metrics.
35	7a3db57ed138532e3fe55bfac07cfa144d539525	2606.27372	10.48550/arXiv.2606.27372	2026	DnA: Denoising Attention for Visual Tasks	R. Campos	5	arXiv.org	0	The softmax activation in multihead attention (MHA) is the de facto standard for attention-based models in visual perception tasks. However, standard softmax can produce noisy attention patterns that dilute relevant features and degrade its performance. In this paper, we propose Denoising Attention or DnA, in which, first, a positive query identifies which image features belong to the correct class, and a negative query identifies closely associated but irrelevant image features. DnA then projects these interactions into two distinct subspaces with larger principal angles, promoting subspace separation and improved discriminability. Using a ViT-B backbone, our proposed DnA achieves an absolute gain of 0.8% on ImageNet-1K compared to the baseline. We further show improvements across multiple visual understanding tasks, including video understanding with video transformers (1.8%) and video LLMs (0.5%). Our extensive empirical analyses justify the design choices involving two interacting subspaces and the denoising effect of DnA.
36	4326407b89fc59dc823ddecf8530d64109b1dd29	2606.24752	10.48550/arXiv.2606.24752	2026	Can Scale Save Us From Plasticity Loss in Large Language Models?	J. Hernandez-Garcia	3	arXiv.org	1	The loss of plasticity - the ability of a network to learn new information after having already learned older information - is a fundamental challenge in creating artificial neural networks capable of continual learning. Although this phenomenon has been known for decades, it has mostly been studied in older, relatively small architectures and rarely in natural-language domains. To determine whether loss of plasticity remains a problem in the modern transformer-based LLM paradigm, we study plasticity loss in GPT-style Transformer models trained on a multilingual continual learning problem. Consistent with prior work, we find evidence of plasticity loss across models ranging from 5M to 314M non-embedding parameters, as measured by deterioration on a held-out Vietnamese probing task. We further find that the onset of plasticity loss follows a predictable scaling law, growing sublinearly with model size. These results suggest that larger models may delay the measurable effects of plasticity loss, but that increasing parameter count alone is likely to be insufficient to completely prevent it. We also find evidence of plasticity loss under stationary multilingual training, challenging the view that the phenomenon is exclusive to continual learning with abrupt task changes. Overall, our results suggest that even large Transformer language models trained on natural-language will eventually lose the ability to efficiently adapt to new data after sufficiently long training, in both continual and stationary settings.
37	a538f5e21803d67b69012214dcf238ddd038bb2f	2606.21974	10.48550/arXiv.2606.21974	2026	Fine-Tuning Large Language Models for Quantum Reasoning	K. Ip	5	arXiv.org	0	Large language models (LLMs) exhibit abilities beyond natural language modelling and text generation. Recent advances in their reasoning capabilities have spurred interest in applying LLMs to complex scientific tasks requiring deep domain expertise and sophisticated reasoning. Quantum computing, as a highly specialised field with significant knowledge barriers and hardware constraints, could greatly benefit from such advancements. However, a key open question that first must be answered is: How can we develop fine-tuning pipelines that instil genuine quantum reasoning in LLMs, rather than task-specific pattern matching? We study this question through quantum circuit simulation as a training objective, where the model must predict the measurement probability distribution resulting from a sequence of quantum gate operations. We propose and compare two fine-tuning pipelines: (1) Supervised Fine-Tuning (SFT) on explicit gate-by-gate state-vector simulation traces, and (2) a two-stage SFT+Group Relative Policy Optimisation (GRPO) approach that sequentially applies SFT followed by GRPO with verifiable rewards. Our findings show that SFT achieves near-perfect in-distribution and gate-count extrapolation accuracy, significantly outperforming both the base model and the GPT-OSS-120B baseline. SFT+GRPO trades some in-distribution precision for better generalisation to larger qubit systems that SFT alone cannot handle. Both pipelines significantly outperform the baselines, demonstrating that targeted fine-tuning on explicit reasoning traces is an effective strategy for advancing quantum reasoning in LLMs.
38	362d213a484b18056f858780e0afcc761d3bc222	2606.16768	10.48550/arXiv.2606.16768	2026	Taming Curvature: Architecture Warm-Up for Stable Transformer Training	Sameera Ramasinghe	9	arXiv.org	1	Training billion-parameter Transformers is often brittle, with transient loss spikes and divergence that waste compute. Even though the recently developed Edge of Stability (EoS) theory provides a powerful tool to understand and control the stability of optimization methods via the (preconditioned) curvature, these curvature-controlling methods are not popular in large-scale Transformer training due to the complexity of curvature estimation. To this end, we first introduce a fast online estimator of the largest (preconditioned) Hessian eigenvalue (i.e., curvature) based on a warm-started variant for power iteration with Hessian-vector products. We show theoretically, and verify empirically, that the proposed method makes per-iteration curvature tracking feasible at billion parameter scale while being more accurate. Using this tool, we find that training instabilities coincide with surges in preconditioned curvature and that curvature grows with depth. Motivated by these observations, we propose architecture warm-up: progressively growing network depth to carefully control the preconditioned Hessian and stabilize training. Experiments on large Transformers validate that our approach enables efficient curvature tracking and reduces instabilities compared to existing state-of-the-art stabilization techniques without slowing down convergence.
39	cacb1f26dd4ca32fb6663fb7ca0489a05b598a6e	2606.08105		2026	A Unifying View of Attention Sinks: From Mechanisms to Architectural Interventions	Lukas Fesser	5		1	When attention concentrates on a single token, a sink, what is the model actually computing? Attention sinks are ubiquitous in softmax transformers, yet this shared visual signature can hide fundamentally different algorithms. We show that visually similar sink patterns can reflect two distinct mechanisms: (i) adaptive nop, where a head suppresses its update by routing to a null token, and (ii) broadcast, where a sink aggregates and redistributes global information. Each mechanism leaves distinct traces (nop-sinks exhibit negligible value norms; broadcast sinks induce low-rank outputs), which we formalize on synthetic tasks and use to derive practical diagnostics. Applied to pretrained vision transformers, these diagnostics reveal that both mechanisms exist at scale: sinks transition from CLS in early layers to patches in deeper layers and concentrate in specialized heads. Causal interventions further connect these signatures to near-null suppression and shared residual contributions. We then use architectural interventions to show how these computations can be reorganized: gating eliminates detected nop-like sinks but increases broadcast-like sinks, registers relocate rather than remove sink computation, and our position-free global pathway provides an explicit route for shared communication that reduces the broadcast-like sinks induced by gating. On dense probes, combining gating with the global pathway gives the strongest results among the tested variants, despite retaining some broadcast-like sinks. Overall, we find that the same attention pattern can reflect two very different computations, and that effective intervention depends not only on identifying the computation, but also on providing architectural alternatives through which the model can reorganize it.
40	adb63e09b6b1a75fabca10e57c0dc61bd49538e3	2608.12332	10.18653/v1/2026.acl-long.1179	2026	Can Spectral-Clipping Enable Better Learning While Forgetting Less for Low-Rank Adaptation?	Hyowon Wi	2	Annual Meeting of the Association for Computational Linguistics	1	In recent years, low-rank adaptation (LoRA) has emerged as a significant paradigm that freezes pre-trained weights and introduces small, learnable adapters instead of fine-tuning the full set of parameters. In this work, we uncover several key insights regarding the singular components of network parameters based on Singular Value Decomposition (SVD). Firstly, the principal singular components with large singular values in pre-trained network parameters can be effectively reused during fine-tuning, whereas the minor components with smaller singular values are more task-specific and require substantial adaptation. Secondly, we first establish the theoretical connection that the uncontrolled growth of singular values in LoRA adapters leads to the forgetting of pre-trained knowledge -- a well-known issue referred to as catastrophic forgetting. Building on these observations, we propose SCLoRA, which injects parameterized singular components with spectral clipping into the pre-trained model in a way that is aware of the spectral distribution of the pre-trained model. SCLoRA effectively adapts to new tasks by focusing updates on components that require adaptation, while simultaneously alleviating catastrophic forgetting. We conduct extensive experiments and demonstrate that SCLoRA not only improves downstream performance but also effectively retains pre-trained knowledge.
41	fcfd6b53ce91e07f609a20de6955b9f2649b1209		10.1109/ICRA57385.2026.11697461	2026	Multi-Horizon Lane Change Maneuver Prediction Using Multi-Modal Transformers	P. Rama	3	IEEE International Conference on Robotics and Automation	0	Predicting lane change maneuvers is essential for ensuring safe autonomous driving, especially in complex urban environments. Building upon prior multi-modal and graph-based approaches, this work introduces a novel transformer-based architecture for multi-horizon lane change prediction that jointly estimates the lane change maneuver and the lane change phase. The proposed model integrates visual information from surround-view cameras, semantic masks for free space and lane markings, interaction-aware graph representations, and ego-vehicle state signals, within a unified transformer framework to capture spatial-temporal dependencies. In addition, a multi-level uncertainty estimation branch quantifies confidence at the level of modality, fusion, and prediction, to enhance interpretability and reliability. Experiments are conducted on WylonSet++, an extended in-house dataset collected using an instrumented test vehicle, annotated for lane change behavior analysis and maneuver phase transitions. The dataset comprises synchronized front-facing camera images, left and right surround-view camera images, together with vehicle state data. The dataset contains approximately 600 lane change sequences, providing the foundation for this study. Extensive evaluations demonstrate strong performance in anticipating lane change maneuvers and phase progression across short-and long-term prediction horizons in diverse real-world traffic scenarios.
42	cdbb19965d15ed55d9b9aaeef14b797205bf6f6a	2605.29983	10.48550/arXiv.2605.29983	2026	Improving Adversarial Robustness of Attribution via Implicit Regularization	Amir Mehrpanah	3	arXiv.org	0	The adversarial robustness of attributions is a fundamental requirement for reliable explainability in deep learning, yet existing approaches typically rely on computationally expensive explicit regularization. In this work, we show that attribution robustness can arise implicitly from the learning dynamics of standard stochastic gradient descent. We theoretically motivate this effect through connections between parameter-space and input-space curvature, and validate it across architectures, datasets, and attribution methods, with negligible computational overhead. In contrast, we prove that such robustness gains often does not transfer to attention-based attribution under softmax normalization, due to inherent entropy constraints, and we validate this limitation experimentally. Finally, we show that replacing softmax attention with kernel-based attention restores the robustness gains in transformer models. Our results highlight learning dynamics as a principled and practical mechanism for robust explainability, and reveal fundamental limitations of attention-based attribution under normalization.
43	2c543c1e741e1b4084f10ec2f9dd52c872986af0		10.3390/ai7060194	2026	The Thermodynamics of Attention: First Law and Landauer Limit Analogues for Learning and Explainability	R. Sotero	2	Applied Informatics	1	The Transformer architecture drives modern Artificial Intelligence (AI), yet the physical principles that may constrain self-attention training remain poorly characterized. We develop a thermodynamic framework for attention training, drawing on the established Boltzmann correspondence between softmax attention and equilibrium statistical mechanics, and we propose a First Law analogue that decomposes the training energy budget into a heat term (the entropic cost of ordering attention) and a work term (the gain in mutual information about the target). From this framework we derive a Landauer-type bound on learning, which states that the loss reduction during training is bounded below by the entropic cost of structuring attention against thermal noise. The bound is satisfied across all configurations tested: 625 grid points spanning three datasets on a compact Vision Transformer trained from scratch (MNIST, CIFAR-10, and OrganAMNIST), and ten temperatures on a pretrained ViT-Small fine-tuned on Food-101. Reusing the same physical principles at inference time, we show that the thermodynamic work performed by each input patch provides a quantitative, energy-based measure of feature importance that outperforms standard attention weights and Integrated Gradients on ImageNet across pretrained ViT-Small, ViT-Base, and ViT-Large (22M to 304M parameters). The result is an integrated diagnostic framework that links phase structure, training-time bounds, and inference-time attribution within a single empirically falsifiable thermodynamic apparatus.
44	88e859cdabb30a713f369f901052d9588a5c0280	2605.25619	10.48550/arXiv.2605.25619	2026	Analogies between Transformer Layers and Power Method	Chenglong Li	2	arXiv.org	0	In the paper we show that there is an analogy between the operations occurring in a layer of a transformer (projections and layer normalizations, disregarding the feedforward neural network) and a step in the power method. Coherently with this analogy, we show that passing through a layer the tokens tend to be tilted towards the principal eigenvector of a matrix which is the product of the output and value weight matrices of that layer. In the special case of a transformer with shared weights (i.e., in which all layers have identical weights) then the alignment with this principal eigenvector is particularly evident empirically, and can also be shown analytically. The analogy also suggests a method to steer the output of the transformer towards an arbitrary desired direction in token space.
45	706c8aafb1df7bbd1d079f20e14e324ea11dde2d	2605.21391	10.48550/arXiv.2605.21391	2026	Post-Hoc Understanding of Metaphor Processing in Decoder-Only Language Models via Conditional Scale Entropy	La. Chakrabarti	6	arXiv.org	0	Metaphor requires a language model to resolve a token whose contextual meaning diverges from its basic literal sense. Understanding how transformer models organize this reinterpretation across depth remains an open problem in mechanistic interpretability. We introduce conditional scale entropy (CSE), a wavelet-derived measure of how broadly transformer computation engages across frequency scales at each layer position. Two theorems establish that CSE is invariant to update magnitude, isolating the structural pattern of updates from their intensity. Using CSE, we find that metaphorical tokens produce significantly higher spectral breadth than literal tokens at contiguous layer positions on every decoder-only architecture tested, from 124M to 20B parameters (GPT-2 family, LLaMA-2 7B, GPT-oss 20B). The effect survives cluster-based permutation correction, recurs in the early-to-mid relative depth range across models, and converges with an independent analysis of 200 naturalistic VUA pairs. Specificity controls further show that the effect is not explained by semantic complexity or by matched propositional content. These results identify multi-scale coordination as a consistent signature of metaphorical language processing in the decoder-only architectures examined, and establish CSE as a principled tool for characterizing cross-depth structure in transformers.
46	211b4e2fd4838a35b9c470e67ac96c67e80f765c	2605.21292		2026	Unbiased Gradients, Moving Stability Boundaries: Exact Mini-Batch Geometry in Linear Self-Attention	K. Balasubramanian	1		0	Unbiased stochastic gradients can match the full gradient in expectation while changing finite-step stability. We study this effect in one-layer linear self-attention for in-context linear regression, where shared-mode mini-batch training reduces exactly to a random two-factor map. The full-batch map preserves an elliptic region, whereas sampled curvature and target correlation create batch-dependent stability boundaries. Their combined fluctuation is centered at the parameter-update level but induces a strictly outward drift of the full-batch boundary coordinate. We derive an exact one-step crossing criterion, adaptive concentration bounds, and conditional drift identities, and prove almost-sure escape in a balanced switching model. For independent long prompts, mode leakage decays at the inverse square root of prompt length. For finite-token softmax attention, we derive an explicit boundary-crossing law governed by an effective context length that shrinks as attention scores concentrate. Experiments in ambient attention objectives verify the reduced predictions, and four trained causal softmax Transformers show larger mean curvature shifts for smaller first-step batches, consistent with the predicted nonlinear outward bias.
47	a4777999a5574f5417b2ebd53b982d1f3047cf57	2605.20314	10.48550/arXiv.2605.20314	2026	Less Data, Faster Training: repeating smaller datasets speeds up learning via sampling biases	Jingwen Liu	4	arXiv.org	0	This work investigates the ``small-vs-large gap'', where repeating on fewer samples can lead to compute saving during training compared to using a larger dataset. This is observed across algorithmic tasks, architectures and optimizers and cannot be explained using prior theory. We argue that the speedup comes from appropriate layer-wise growth enabled by sampling biases, which is more pronounced when the dataset size is smaller. We provide both theoretical analysis and empirical evidence from various interventions. Our results suggest that using a smaller dataset with more repetitions is not just a fallback strategy under data scarcity, but can be proactively leveraged as a favorable inductive biases for optimization, particularly in reasoning tasks.
48	0d6cc4323300151f5343043a2d8c813ae0931c83	2605.10504	10.48550/arXiv.2605.10504	2026	Learning Less Is More: Premature Upper-Layer Attention Specialization Hurts Language Model Pretraining	Jin-Chang Zhu	6	arXiv.org	2	A causal-decoder block is hierarchical: lower layers build the residual basis that upper layers attend over. We identify a failure mode in GPT pretraining: upper layers commit to sharp attention patterns before lower-layer features stabilize. We call this premature upper-layer attention specialization. Temporarily slowing only upper-layer Q/K projections during early training improves final perplexity and downstream accuracy without altering other parameters; it prevents upper attention from collapsing onto an immature residual basis. In LLaMA-style blocks, the same intervention is nearly unnecessary. Through ablations, we isolate multiplicative gated FFNs (not RMSNorm or bias removal) as the component that suppresses the upstream residual writes driving the failure. A pathwise analysis unifies both findings: the learning-rate intervention reduces a step-size factor, while gated FFNs reduce a residual-energy factor on the same growth pathway. Our results identify upper-layer Q/K timing as a concrete interaction point between decoder architecture and optimization.
49	aa92feea55fc95a645f5e2db346ec12bb2e1b64f		10.1371/journal.pone.0348610	2026	A multi-view TSK fuzzy system with deformable Gaussian membership functions and rule-level attention for classification	Zhi-Qi Huang	3	PLoS ONE	1	This study presents a novel multi-view TSK fuzzy system that integrates deformable Gaussian membership functions with a rule-level attention mechanism (MDA-TSK-FS), aiming to improve the modeling capacity and flexibility of fuzzy systems in high-dimensional and complex classification tasks. In the antecedent part, learnable deformation offsets are introduced, enabling the membership function centers of each fuzzy rule to dynamically adjust according to data characteristics. This design enhances the adaptability of rules to the input space. Furthermore, a multi-head attention mechanism at the rule level is incorporated to adaptively allocate rule weights based on sample-specific information, thereby enabling dynamic modeling of rule importance and optimized rule selection. Extensive experiments on five public multi-view datasets, including Caltech7, Handwritten, Dermatology, Forest, and EEG, demonstrate that the proposed model consistently achieves superior performance, reaching classification accuracies of 94.38%, 98.62%, 98.58%, 88.57%, and 69.75%, respectively, and outperforming strong baselines. Ablation studies further verify the effectiveness of the two core components: the deformable antecedent structure and the rule-level attention mechanism, which individually improved EEG classification accuracy by approximately 7% and 6%, and jointly by 9.25% compared to the baseline. Notably, the model exhibits superior generalization and interpretability, particularly when processing multi-source heterogeneous data. These findings indicate that the proposed approach provides a new modeling paradigm for multi-view fuzzy inference, offering both theoretical contributions and practical application potential.
50	7dabea735c068c35aa76023e6c0105d5d77b51dc	2605.09549	10.48550/arXiv.2605.09549	2026	When Adaptation Fails: A Gradient-Based Diagnosis of Collapsed Gating in Vision-Language Prompt Learning	Yu Fang	3	arXiv.org	0	Adaptive prompting mechanisms have been proposed to enhance vision-language models by dynamically tailoring prompts to inputs. However, in frozen few-shot prompt learning with CLIP-style backbones, we systematically observe that adaptive gates and prompt-selection modules often collapse: they produce nearly constant outputs, contribute negligible gradient signals, and frequently fail to outperform fixed prompts. To further explore this issue, we present a systematic diagnostic study to uncover the underlying causes and conditions of adaptation failure. Through controlled experiments across datasets and multiple prompt learning architectures, we identify two recurring failure modes: gradient magnitude imbalance and gate degradation. Our findings invite a re-examination of indiscriminately adding architectural complexity in parameter-efficient learning and clarify when prompt-level adaptive gating is, and is not, effective in this regime.
51	676331eb93613b6999e55ad0a9cb0fafc881cbe3	2605.09271	10.48550/arXiv.2605.09271	2026	Shaping Schema via Language Representation as the Next Frontier for LLM Intelligence Expanding	Zhiqin Yang	7	arXiv.org	2	Although natural language is the default medium for Large Language Models (LLMs), its limited expressive capacity creates a profound bottleneck for complex problem-solving. While recent advancements in AI have relied heavily on scaling, merely internalizing knowledge does not guarantee its effective application. Defining language representation as the linguistic and symbolic constructs used to map and model the real world, this paper argues that shaping schemas through advanced language representation is the next frontier for expanding LLM intelligence. We posit that an LLM's knowledge activation and organization -- its schema -- depends heavily on the structural and symbolic sophistication of the language used to represent a given task. This paper contributes both a formalization of this claim and the empirical evidence to support it. With a new formalization, we present multiple lines of evidence to support our position: Firstly, we review recent empirical practices and emerging methodologies that demonstrate the substantial performance gains achievable through deliberate language representation design, even without modifying model parameters or scale. Secondly, we conduct controlled experiments showing that LLM performance and its internal feature activations vary under different language representations of the same underlying task. Together, these findings highlight language representation design as a promising direction for future research.
52	333829d7f3bfda7404312869ccdcb4a5a290531e	2605.08578	10.48550/arXiv.2605.08578	2026	Probing the Impact of Scale on Data-Efficient, Generalist Transformer World Models for Atari	J. Kim	1	Trans. Mach. Learn. Res.	0	Developing generalist systems that retain human-like data efficiency is a central challenge. While world models (WMs) offer a promising path, existing research often conflates architectural mechanisms with the independent impact of model \emph{scale}. In this work, we use a minimalist transformer world model to analyze scaling behaviors on the Atari 100k benchmark, using fixed offline datasets derived from a presupposed expert policy. Our results reveal that environments fundamentally fall into distinct scaling regimes, even when constrained by identical offline data budgets and model capacities. For individual tasks, some environments naturally allow models to pass the interpolation threshold, yielding monotonic improvements in the overparameterized regime, while others remain trapped in the classical regime, where larger world models degrade fidelity. In the unified setting, i.e., a single transformer trained on a suite of 26 Atari environments, we uncover that joint training stabilizes scaling dynamics, ensuring monotonic gains across all environments, regardless of their distinct inherent scaling regimes. Finally, we demonstrate that improved fidelity translates directly to downstream control, with policies learned entirely within the simulated dynamics achieving a median expert-random-normalized score of 0.770. Our findings suggest that future progress lies as much in precise scaling strategies as in architectural innovation.
53	83396bcd2b69e5b17e5cb915484010d7823ac1bc	2605.06169	10.48550/arXiv.2605.06169	2026	Mean Mode Screaming: Mean-Variance Split Residuals for 1000-Layer Diffusion Transformers	Pengqian Lu	1	arXiv.org	0	Scaling Diffusion Transformers (DiTs) to hundreds of layers introduces a structural vulnerability: networks can enter a silent, mean-dominated collapse state that homogenizes token representations and suppresses centered variation. Through mechanistic auditing, we isolate the trigger event of this collapse as Mean Mode Screaming (MMS). MMS can occur even when training appears stable, with a mean-coherent backward shock on residual writers that opens deep residual branches and drives the network into a mean-dominated state. We show this behavior is driven by an exact decomposition of these gradients into mean-coherent and centered components, compounded by the structural suppression of attention-logit gradients through the null space of the Softmax Jacobian once values homogenize. To address this, we propose Mean-Variance Split (MV-Split) Residuals, which combine a separately gained centered residual update with a leaky trunk-mean replacement. On a 400-layer single-stream DiT, MV-Split prevents the divergent collapse that crashes the un-stabilized baseline; it tracks close to the baseline's pre-crash trajectory while remaining substantially better than token-isotropic gating methods such as LayerScale across the full schedule. Finally, we present a 1000-layer DiT as a scale-validation run at boundary scales, establishing that the architecture remains stably trainable at extreme depth.
54	8a8e30d43370ba4bd1dee85f4f54f2ad31414735	2605.04893	10.48550/arXiv.2605.04893	2026	Self-Attention as Transport: Limits of Symmetric Spectral Diagnostics	D. Dahlem	3	arXiv.org	0	Every attention head defines a degree-normalized transport operator, and a growing family of diagnostics reads model behavior (hallucination among them) from its spectrum. We ask what such diagnostics can and cannot infer. The operator splits orthogonally into a symmetric part governing transport \emph{capacity} and an antisymmetric part encoding \emph{orientation}. We prove an identifiability limit: every transpose-invariant spectral diagnostic is \emph{orientation-blind} (unable to distinguish an operator from its transpose, hence blind to the orientation of information flow), with a transpose-stability bound limiting any Lipschitz diagnostic's transpose sensitivity by the asymmetry coefficient $G$. This bounds what spectral diagnostics of the attention operator can resolve (e.g.\ LapEigvals and the attention-spectral branch of LLM-Check). On the surviving axis, a closed-form bipartite-Cheeger landscape shows uniform causal attention obeys an $n$-independent \emph{temporal-cut} floor $\phi \ge 1/5$ while window attention pierces it as $O(w/n)$; the floor is an idealized benchmark, not an empirical attractor, and the fraction of real heads falling below it is itself an empirically stable architectural descriptor. The two-axis diagnostic ($\phi$ for capacity, $G$ for asymmetry magnitude) yields a falsifiable polarity prediction, borne out \emph{in sign} under length-controlled, forced-scoring evaluation across decoder-only, encoder-only, and encoder--decoder models (capacity-axis signal 0.62--0.84 LC-AUROC): polarity reverses between HaluEval and MedHallu, directionally as predicted though asymmetric in strength, with decision polarity calibrated per regime.
55	ac162ed751574641cb16057b7b87cde4d6c88c2c	2605.04279	10.48550/arXiv.2605.04279	2026	Gradient Flow Structure and Quantitative Dynamics of Multi-Head Self-Attention	Ayan Pendharkar	1	arXiv.org	2	Transformer self-attention can be interpreted as a gradient flow on the unit sphere, in which tokens evolve under softmax interaction potentials and tend to form clusters. While prior work has established clustering behavior for single-head attention, the multi-head setting remains less understood due to geometric interference between heads, which invalidates standard monotonicity arguments. In this work, we develop a theoretical framework for multi-head self-attention dynamics and resolve several open questions. We show that, under suitable conditions on the score matrices, a natural multi-head energy functional is non-decreasing along both flat and spherical dynamics. We identify the key obstruction to per-head monotonicity as radial shadow terms, which are projections of each head's output onto token directions, persisting even under orthogonality assumptions. We introduce a sufficient condition ensuring monotonicity and establish robustness to approximate orthogonality. In a simplified scalar-head regime with equiangular token configurations, we derive a closed-form expression for the critical inverse temperature governing clustering behavior, and show that heterogeneous heads exhibit super-additive clustering rates. In this regime, we also prove a separation in clustering time between ReLU and softmax attention in the linearized dynamics. Finally, we establish an entropy production identity and show that attention entropy increases monotonically toward equilibrium as clustering progresses. Our results provide a unified perspective on the dynamics of multi-head attention and clarify the mechanisms underlying clustering and stability in transformer models.
56	748bfc4fecbe0584dd6bca0c445e87273df7a265	2604.28118		2026	Hierarchical Fault Detection and Diagnosis for Transformer Architectures	Sigma Jahan	4		0	Transformers now underpin critical AI systems across industry and research. Yet their faults can silently alter model behavior without runtime errors, and existing techniques offer little support for tracing these failures to their component and root cause. Such faults evade detection because loss and numerical values stay normal, and the visible symptom rarely identifies the component responsible. We present DEFault++, a hierarchical learning-based technique that first detects a fault, then identifies the affected component, and finally the cause within it, helping developers effectively debug transformer models. DEFault++ organizes component-level runtime measurements with a Fault Propagation Graph (FPG), a structural prior over the architecture's dependency paths, and reports the evidence behind each diagnosis. To train and evaluate it, we construct DEFault-bench, a benchmark of 5,556 labeled runs from mutation testing across seven models, nine tasks, and both encoder and decoder architectures. DEFault++ improves fault detection over four prior techniques, reaching an F1 of 0.826--0.909, and in a developer study with 21 participants, it raises repair accuracy from 57.1% to 83.3%. These results show that transformer fault diagnosis benefits from component-level measurements and architecture-aware reasoning rather than model-level behavior alone, and DEFault-bench provides a foundation for further research on transformer fault diagnosis.
57	b83f29027e5e6fc9fd2235843681335b2b150ae7		10.1145/3811409	2026	Understanding Hallucinations in Large Visual and Language Models	Zheng-Yi Ho	3	ACM Computing Surveys	3	The rapid deployment of large language and vision models in real-world applications has intensified the need to address hallucinations—instances where models generate incorrect or incoherent outputs. These failures can spread misinformation and degrade workflows, causing financial and operational harm. Despite extensive research efforts, our understanding of hallucinations remains limited and fragmented. Without clear understanding, solutions risk addressing disparate symptoms rather than root causes, which undermines their effectiveness and generalisability during deployment. To address this, we first introduce a unified, multi-level framework to characterise both image and text hallucinations across broad applications, helping reduce conceptual fragmentation. Then, we trace their root causes to identifiable mechanisms within a model’s lifecycle in a task-modality interleaved manner, fostering a deeper and more holistic understanding. Our investigations reveal hallucinations as predictable consequences of underlying distributions and biases. By enhancing our understanding of hallucinations, this survey lays the groundwork for more effective solutions to hallucinations in generative AI systems.
58	76c19270fcdb57581ec72f6b182715f56c864373		10.3390/quantum8020036	2026	Bipolar Entropy vs. Entropy/Negentropy: From Quantum Emergence to Agentic AI&QI with Collectively Entangled Bipolar Strings ER ≥≥ EPR	Wen-Ran Zhang	2	Quantum Reports	0	While the quantum emergence of spacetime is becoming a major research topic in physics, the quantum emergence of intelligence has not been widely researched in quantum information science (QIS). Following causal-logical quantum gravity theory, bipolar entropy vs. entropy and negative entropy (or negentropy) are reviewed and distinguished for quantum emergence/submergence of quantum agent (QA) and quantum intelligence (QI) in algebraic terms. This work refers to QA as an entangled bipolar string/superstring in bipolar dynamic equilibrium (BDE) and QI being centered on logically definable causality in regularity, mind-light-matter unity, and brain-universe similarity. ER = EPR is extended to ER ≥≥ EPR for the mathematical scalability of bipolar strings and their collective entanglement. The extension leads to a number of conjectures, testable predictions, and theorems. The term “equilibraton” is proposed as a type of EPR or bipolar generic string to serve as an entropic stitch to collectively hold the universe together as a quantum entanglement in BDE with ubiquitous, regulated local emergence and submergence of QA&QI. Equilibraton leads to the concept of bipolar entropy square—a complete entropic solution to the background issue in quantum gravity. With complete background independence, energy/information conservational bipolar entropy, energy/information invariance, bipolar entropy non-additivity, and equilibrium-based plateau concavity are introduced. The nature of the one-dimensional arrow of time is conjectured. As a unification of order and disorder for equilibrium-based regulation, bipolar entropy bridges QA&QI to agentic AI, where quantum-bio-economics can be viewed as a topological intervention of a natural dynamic equilibrium in a social or natural world. Use cases are reviewed to illustrate the practical and theoretical aspects of bipolar entropy in business management, quantum-bio-economics, quantum cryptography, physics, and biology. Eddington–Einstein’s comments on entropy are revisited. It is expected that bipolar entropy will bring quantum emergence/submergence to agentic AI&QI for entangled machine thinking and imagination as a naturally scalable and testable foundation of real-world quantum gravity, quantum information science (QIS), quantum cognition, and quantum biology (QCQB) to enhance Large Language AI Models (LLMs) and machine intelligence.
59	31979d6d59112f38023fd7c052b0ec176f8e73ee	2604.17324		2026	Capacity-Controlled Global Attention for Graph Transformers	Yang Liu	6		1	Global self-attention drives modern graph transformers, yet the softmax at its core imposes a structural constraint rarely examined directly: every attention row is non-negative and sums to one, so each per-head output is a mass-conserving convex combination of value vectors. A node can never"attend to nothing."We argue this conservation constraint is a single root cause behind three pathologies usually studied in isolation: the collapse of node representations with depth (over-smoothing), a low-rank bottleneck on per-head outputs, and brittle optimization in deep stacks. Drawing on how sigmoid gating removes analogous attention sinks in language models, we introduce SigGate-GT, a graph transformer that applies a learned, per-head, input-conditioned sigmoid gate to the attention output inside the GraphGPS framework. The gate is a smooth, per-dimension"volume control"that can drive head outputs toward zero, relaxing the constraint without abandoning attention's probabilistic interpretation. Analytically and through synthetic experiments, we show the gate strictly increases the stable rank of per-head outputs, and connect this rank gain to all three manifestations. On five molecular and long-range benchmarks, SigGate-GT matches the prior best on ZINC (0.059 MAE), records the strongest result among the graph-transformer baselines we evaluate on ogbg-molhiv (82.47% ROC-AUC), and is competitive on ogbg-molpcba and the Long-Range Graph Benchmark, with statistically significant gains over GraphGPS on all five datasets (p<0.05). Mechanism analyses confirm the diagnosis: gating slows over-smoothing (a 30% mean relative gain in representation diversity across 4-16 layers), keeps attention entropy from collapsing, and stabilizes training across a 10x learning-rate range, at about 1% parameter overhead on OGB and under 3% wall-clock cost.
60	71835f690f33e609099c31ccd249805ee1edb614	2604.15180	10.48550/arXiv.2604.15180	2026	AdaSplash-2: Faster Differentiable Sparse Attention	Nuno Gonçalves	7	arXiv.org	4	Sparse attention has been proposed as a way to alleviate the quadratic cost of transformers, a central bottleneck in long-context training. A promising line of work is $\alpha$-entmax attention, a differentiable sparse alternative to softmax that enables input-dependent sparsity yet has lagged behind softmax due to the computational overhead necessary to compute the normalizer $\tau$. In this paper, we introduce AdaSplash-2, which addresses this limitation through a novel histogram-based initialization that reduces the number of iterations needed to compute $\tau$ to typically 1--2. The key idea is to compute a coarse histogram of attention scores on the fly and store it in on-chip SRAM, yielding a more accurate initialization that enables fast forward and backward computation. Combined with a sparsity-aware GPU implementation that skips zero blocks with low overhead, AdaSplash-2 matches or improves per-step training time relative to FlashAttention-2 when block sparsity is moderate-to-high (e.g., $>$60\%), which often occurs at long-context lengths. On downstream tasks, models trained with our efficient $\alpha$-entmax attention match softmax baselines at short-context lengths and achieve substantial gains in long-context settings.
61	cbb04421705c0f05c48f6d49e1e937e3a3d347a7	2604.13123		2026	Spectral Entropy Collapse as a Phase Transition in Delayed Generalisation: An Interventional and Predictive Framework for Grokkin	Xuan Khanh Truong	4		2	Grokking - the delayed transition from memorisation to generalisation in neural networks - remains poorly understood. We study this phenomenon through the geometry of learned representations and identify a consistent empirical signature preceding generalisation: collapse of the spectral entropy of the representation covariance matrix. Across modular arithmetic tasks and multiple random seeds, spectral entropy decreases gradually during training and crosses a stable task-specific threshold before test accuracy rises. A representation-mixing intervention that delays this collapse also delays grokking, including under norm-matched controls, indicating that the effect is not explained by parameter norm alone. We further show that the entropy gap predicts the remaining time until grokking with useful out-of-sample accuracy. To probe the structure underlying this transition, we introduce a Fourier-alignment observable for cyclic-group tasks. Entropy collapse is strongly coupled to the emergence of Fourier-aligned representations, suggesting that spectral entropy tracks concentration of the representation into task-structured directions rather than generic compression alone. The same qualitative dynamics appear in non-abelian group composition tasks, while MLP controls show that entropy collapse by itself is insufficient for grokking in the absence of appropriate inductive bias. Taken together, the results support a view of grokking as a representational phase transition with an observable geometric signature. We discuss the scope and limitations of this interpretation, connections to recent feature-learning and spectral-dynamics work, and directions for testing whether similar transitions appear in larger-scale learning systems.
62	1c528c5c9bbc23b791e55c7af736e5d1283ecf43	2604.11890	10.48550/arXiv.2604.11890	2026	Subcritical Signal Propagation at Initialization in Normalization-Free Transformers	Sergey Alekseev	1	arXiv.org	1	We study signal propagation at initialization in transformers through the averaged partial Jacobian norm (APJN), a measure of gradient amplification across layers. We extend APJN analysis to transformers with bidirectional attention and permutation-symmetric input token configurations by deriving recurrence relations for activation statistics and APJNs across layers. Our theory predicts how attention modifies the asymptotic behavior of the APJN at large depth and matches APJNs measured in deep vision transformers. The criticality picture known from residual networks carries over to transformers: the pre-LayerNorm architecture exhibits power-law APJN growth, whereas transformers with LayerNorm replaced by elementwise $\tanh$-like nonlinearities have stretched-exponential APJN growth, indicating that the latter are subcritical. Applied to Dynamic Tanh (DyT) and Dynamic erf (Derf) transformers, the theory explains why these architectures can be more sensitive to initialization and optimization choices and require careful tuning for stable training.
63	5f6d5c615c711ca57db31bae21dcf75af94ce5e2	2604.07925	10.48550/arXiv.2604.07925	2026	Sinkhorn doubly stochastic attention rank decay analysis	M. Lapenna	3	arXiv.org	2	The self-attention mechanism is central to the success of Transformer architectures. However, standard row-stochastic attention has been shown to suffer from significant signal degradation across layers. In particular, it can induce rank collapse, resulting in increasingly uniform token representations, as well as entropy collapse, characterized by highly concentrated attention distributions. Recent work has highlighted the benefits of doubly stochastic attention as a form of entropy regularization, promoting a more balanced attention distribution and leading to improved empirical performance. In this paper, we study rank collapse across network depth and show that doubly stochastic attention matrices normalized with Sinkhorn algorithm preserve rank more effectively than standard softmax row-stochastic ones. As previously shown for softmax, skip connections are crucial to mitigate rank collapse. We empirically validate this phenomenon on both sentiment analysis and image classification tasks. Moreover, we derive a theoretical bound for the pure self-attention rank decay when using Sinkhorn normalization and find that rank decays to one doubly exponentially with depth, a phenomenon that has already been shown for softmax.
64	b787d65fb25f135dbcbf0c404c1935fc8a935070		10.1109/TMI.2026.3682009	2026	StableMIL: Entropy-Stabilized Attention-Based Multiple Instance Learning for Morphologically Variable Whole Slide Images	Yi-Nuo Lu	7	IEEE Transactions on Medical Imaging	0	Aggregating features of tens of thousands of patches into Whole Slide Images (WSIs) representations via aggregators is a crucial step in computational pathology. However, existing aggregation strategies overlook the morphological variability of tissue regions in WSIs stemming from differences in clinical procedures and tumor characteristics, leading to two critical limitations: 1) attention collapse in long sequences caused by significant variation in patch numbers across WSIs (ranging from thousands to tens of thousands per WSI); 2) attention misallocation due to under-trained positional embeddings resulting from the non-uniform spatial coordinates introduced by irregular patch distributions. Consequently, current attention-based methods struggle to generalize across this morphological variability, resulting in inconsistent aggregation performance and compromised model reliability in clinical settings. To address these issues, we propose a Entropy-Stabilized Attention-based Multiple Instance Learning (StableMIL) framework, which incorporates an entropy-stabilized attention mechanism to ensure consistent aggregation across WSIs with varying patch numbers and a Randomly Projected 2D rotary position embedding to enhance spatial representation robustness across irregular patch distributions. Extensive theoretical and experimental analyses on nine WSI datasets spanning diverse cancer types, across both classification and survival prediction tasks, demonstrate that StableMIL effectively overcomes the challenges of handling long instance sequences and out-of-distribution spatial coordinates. Our framework consistently outperforms representative baselines, particularly in survival prediction, with stable improvements observed across all evaluated cancer types and morphological scenarios, highlighting its potential for real-world clinical applications. Our source code is available at https://github.com/theeeqi/stableMIL
65	6231dc17ae4114751e4b9b05a0aa2e4aee3b5334		10.1093/bioinformatics/btag218	2026	CPS: mapping physical coordinates to high-fidelity spatial transcriptomics via privileged multi-scale context distillation	Lei Zhang	5	bioRxiv	0	Motivation Spatial transcriptomics enables the dissection of tissue heterogeneity within native contexts, yet current platforms are inherently constrained by high sparsity and low signal-to-noise ratios that obscure fine-grained biological signals. Current efforts to recover these signals are limited by image registration dependencies or the inherent context-blindness of implicit neural representations. Results We introduce the Cell Positioning System (CPS), a context-aware implicit neural representation framework designed to map physical coordinates to high-fidelity spatial transcriptomics via a privileged multi-scale context distillation strategy. CPS treats multi-scale tissue niches as privileged information, employing a teacher network equipped with a multi-scale niche attention mechanism to capture adaptive biological interactions during training. This structural knowledge is explicitly distilled into a student coordinate network, enabling the generation of context-aware expression landscapes solely from spatial coordinates during inference. Benchmarking on the DLPFC dataset demonstrates that CPS achieves state-of-the-art performance in spatial and gene expression imputation and denoising. Furthermore, CPS enables super-resolution to recover high-resolution mouse brain anatomical details and offers interpretability by identifying the scale effective size of biological interactions within human breast cancer tissues. Finally, the framework exhibits superior scalability for large-scale datasets with linear computational complexity. Availability Software is available online at https://github.com/tju-zl/CPS.
66	6e5d15491f05036d7c17dd72a75d2e0a9a2c6b54	2604.03803	10.48550/arXiv.2604.03803	2026	Rényi Attention Entropy for Patch Pruning	Hiroaki Aizawa	2	International Conference on Pattern Recognition	0	Transformers are strong baselines in both vision and language because self-attention captures long-range dependencies across tokens. However, the cost of self-attention grows quadratically with the number of tokens. Patch pruning mitigates this cost by estimating per-patch importance and removing redundant patches. To identify informative patches for pruning, we introduce a criterion based on the Shannon entropy of the attention distribution. Low-entropy patches, which receive selective and concentrated attention, are kept as important, while high-entropy patches with attention spread across many locations are treated as redundant. We also extend the criterion from Shannon to R\'enyi entropy, which emphasizes sharp attention peaks and supports pruning strategies that adapt to task needs and computational limits. In experiments on fine-grained image recognition, where patch selection is critical, our method reduced computation while preserving accuracy. Moreover, adjusting the pruning policy through the R\'enyi entropy measure yields further gains and improves the trade-off between accuracy and computation.
67	b6f5bdcfe2c7c90f3b7e5ba6dd0bef724f22f28c	2604.01683		2026	Invertible Query-Key Coupling Composes with Attention Mechanisms	Barak Gahtan	2		0	Scaled dot-product attention forms its queries and keys as independent linear projections, so the two never interact before the dot product that scores them. We study coupled query-key dynamics, a pre-scoring transformation that evolves each token's query and key jointly through a shared invertible coupling before standard scoring. We realize it as an alternating affine map in the style of real non-volume-preserving flows: the coupling is the identity at initialization, adds a small fraction of parameters per head, and leaves the softmax and surrounding architecture unchanged. We place coupling on top of existing attention methods rather than replacing them, and ask whether that composition helps. On WikiText-103, adding coupling to Differential Attention improves on it at both 150M and 455M parameters. At 455M the gain is significant at sequence length 512 (p=0.003, six seeds), survives a Bonferroni correction and replicates on a held-out test split; it also holds across rotary-embedding training lengths 512, 1024, and 2048. The same additive direction appears when coupling is added to query-key normalization (significant at 150M) and Multi-Token Attention (directional). Matched controls attribute the gain to the joint pre-scoring coupling rather than to added capacity, and show that removing the invertibility guarantee preserves the 455M gain yet is far worse than the base method at 150M, so invertibility is what makes the coupling reliable across scales. On its own, coupling lowers perplexity at 60M and 150M (one-sided Welch tests, p<0.05) but the gain narrows with scale and is not significant at 455M. We relate the construction to the expressivity of coupling flows, use an associative-recall study to map where coupling helps and where it degrades sharp retrieval, and conclude that coupling is most useful in composition with a scoring-stage method rather than as a standalone change.
68	4b12b944d46ebd0cb8719229daeec5a9982f2dce	2604.00726	10.1109/CCGrid68966.2026.00025	2026	Exploring Silent Data Corruption as a Reliability Challenge in LLM Training	A. Altenbernd	3	IEEE/ACM International Symposium on Cluster, Cloud and Internet Computing	5	As Large Language Models (LLMs) scale in size and complexity, the consequences of failures during training become increasingly severe. A major challenge arises from Silent Data Corruption (SDC): hardware-induced faults that bypass system-level detection mechanisms. SDC may behave like benign numerical noise, but can also cause harmful gradient corruption that leads to loss spikes, divergence, or stalled progress. This work provides a controlled study of how intermittent SDC affects LLM pretraining. Using targeted fault injection at the level of GPU matrix-multiply instructions, we characterize the sensitivity of different bit positions, kernel functions, and execution stages. Our analysis shows that locally originating faults can produce impactful corruption, including NaN propagation, short-lived spikes in loss, gradient norm, and attention logits, as well as persistent parameter divergence. Building on the observed corruption signatures, we propose a lightweight detection method that identifies potentially harmful parameter updates. Experiments on LLaMA models with $60 \mathrm{M}, 350 \mathrm{M}$, and 1.3B parameters demonstrate that recomputing the most recent training step upon detection can effectively mitigate the impact of these events.
69	6c8934416d0b2df8bf78b392316062ffddb2c4b4	2604.00199	10.48550/arXiv.2604.00199	2026	QUEST: A robust attention formulation using query-modulated spherical attention	Hariprasath Govindarajan	4	arXiv.org	1	The Transformer model architecture has become one of the most widely used in deep learning and the attention mechanism is at its core. The standard attention formulation uses a softmax operation applied to a scaled dot product between query and key vectors. We explore the role played by norms of the queries and keys, which can cause training instabilities when they arbitrarily increase. We demonstrate how this can happen even in simple Transformer models, in the presence of easy-to-learn spurious patterns in the data. We propose a new attention formulation, QUEry-modulated Spherical aTtention (QUEST), that constrains the keys to a hyperspherical latent space, while still allowing individual tokens to flexibly control the sharpness of the attention distribution. QUEST can be easily used as a drop-in replacement for standard attention. We focus on vision applications while also exploring other domains to highlight the method's generality. We show that (1) QUEST trains without instabilities and (2) produces models with improved performance (3) that are robust to data corruptions and adversarial attacks.
70	c025308b9dfc8741db4a65e9c14222e02dc51d78	2603.29206	10.48550/arXiv.2603.29206	2026	Route-Induced Density and Stability (RIDE): Controlled Intervention and Mechanism Analysis of Routing-Style Meta Prompts on LLM Internal States	Di Zhang	3	arXiv.org	0	Routing is widely used to scale large language models, from Mixture-of-Experts gating to multi-model/tool selection. A common belief is that routing to a task ``expert''activates sparser internal computation and thus yields more certain and stable outputs (the Sparsity--Certainty Hypothesis). We test this belief by injecting routing-style meta prompts as a textual proxy for routing signals in front of frozen instruction-tuned LLMs. We quantify (C1) internal density via activation sparsity, (C2) domain-keyword attention, and (C3) output stability via predictive entropy and semantic variation. On a RouterEval subset with three instruction-tuned models (Qwen3-8B, Llama-3.1-8B-Instruct, and Mistral-7B-Instruct-v0.2), meta prompts consistently densify early/middle-layer representations rather than increasing sparsity; natural-language expert instructions are often stronger than structured tags. Attention responses are heterogeneous: Qwen/Llama reduce keyword attention, while Mistral reinforces it. Finally, the densification--stability link is weak and appears only in Qwen, with near-zero correlations in Llama and Mistral. We present RIDE as a diagnostic probe for calibrating routing design and uncertainty estimation.
71	71c43cfc5311fc71f683a6fc5701f38806a496a4	2604.00243	10.48550/arXiv.2604.00243	2026	UCell: rethinking generalizability and scaling of bio-medical vision models	Nicholas Kuang	3	arXiv.org	0	The modern deep learning field is a scale-centric one. Larger models have been shown to consistently perform better than smaller models of similar architecture. In many sub-domains of biomedical research, however, the model scaling is bottlenecked by the amount of available training data, and the high cost associated with generating and validating additional high quality data. Despite the practical hurdle, the majority of the ongoing research still focuses on building bigger foundation models, whereas the alternative of improving the ability of small models has been under-explored. Here we experiment with building models with 10-30M parameters, tiny by modern standards, to perform the single-cell segmentation task. An important design choice is the incorporation of a recursive structure into the model's forward computation graph, leading to a more parameter-efficient architecture. We found that for the single-cell segmentation, on multiple benchmarks, our small model, UCell, matches the performance of models 10-20 times its size, and with a similar generalizability to unseen out-of-domain data. More importantly, we found that ucell can be trained from scratch using only a set of microscopy imaging data, without relying on massive pretraining on natural images, and therefore decouples the model building from any external commercial interests. Finally, we examined and confirmed the adaptability of ucell by performing a wide range of one-shot and few-shot fine tuning experiments on a diverse set of small datasets. Implementation is available at https://github.com/jiyuuchc/ucell
72	7a8825e69ba4295039e3c40b77c994b4864a5779		10.1007/s41748-026-01132-4	2026	START: A Hybrid Spatio-Temporal Attention ResNet Transformer for Explainable Multivariable Meteorological Bias-correction	Deveshwar Singh	3	Earth Systems and Environment	0
73	90b9f5c9637a3a2c5a88ea69dced672c9c08f043	2603.23998	10.48550/arXiv.2603.23998	2026	Sparse Growing Transformer: Training-Time Sparse Depth Allocation via Progressive Attention Looping	Yao Chen	12	Annual Meeting of the Association for Computational Linguistics	5	Existing approaches to increasing the effective depth of Transformers predominantly rely on parameter reuse, extending computation through recursive execution. Under this paradigm, the network structure remains static along the training timeline, and additional computational depth is uniformly assigned to entire blocks at the parameter level. This rigidity across training time and parameter space leads to substantial computational redundancy during training. In contrast, we argue that depth allocation during training should not be a static preset, but rather a progressively growing structural process. Our systematic analysis reveals a deep-to-shallow maturation trajectory across layers, where high-entropy attention heads play a crucial role in semantic integration. Motivated by this observation, we introduce the Sparse Growing Transformer (SGT). SGT is a training-time sparse depth allocation framework that progressively extends recurrence from deeper to shallower layers via targeted attention looping on informative heads. This mechanism induces structural sparsity by selectively increasing depth only for a small subset of parameters as training evolves. Extensive experiments across multiple parameter scales demonstrate that SGT consistently outperforms training-time static block-level looping baselines under comparable settings, while reducing the additional training FLOPs overhead from approximately 16--20% to only 1--3% relative to a standard Transformer backbone.
74	1ca7cbd482a98133f991dc023163570ca35c5f95		10.1117/12.3107769	2026	Research on the security protection mechanism of transformer-based power data mining facing sequence sensitivity	Yue Chen	4	International Conference on Algorithms, High Performance Computing, and Artificial Intelligence	0	This study aims to address the issue of the Transformer model being vulnerable to adversarial attacks due to its sequence sensitivity in power load forecasting. By defining attention entropy, Top-k weights, and quantification metrics, the study analyzes the model's excessive reliance on key time points. A weighted projection gradient descent attack algorithm is proposed to specifically expose the model's vulnerabilities. Additionally, a defense mechanism based on adversarial training is constructed, integrating regularization through attention distribution diversity. Experimental results show that the defense model achieves a prediction accuracy of RMSE 45.3 MW and MAPE 3.2%, outperforming the standard model. Under a strong attack with ε=0.05, the performance drop is controlled within 20%, while attention entropy increases from 2.31 to 3.15, and the Top-5 weight decreases from 0.72 to 0.58. This confirms that the defense mechanism significantly enhances robustness by dispersing the attention distribution. The conclusion indicates that this method fundamentally improves model security while maintaining accuracy, providing an effective security protection solution for power data mining systems.
75	38206fd467777e7c9f1ef2672145093a96338222	2603.14483	10.48550/arXiv.2603.14483	2026	Disentangling Dynamical Systems: Causal Representation Learning Meets Local Sparse Attention	M. Baumgartner	4	arXiv.org	3	Parametric system identification methods estimate the parameters of explicitly defined physical systems from data. Yet, they remain constrained by the need to provide an explicit function space, typically through a predefined library of candidate functions chosen via available domain knowledge. In contrast, deep learning can demonstrably model systems of broad complexity with high fidelity, but black-box function approximation typically fails to yield explicit descriptive or disentangled representations revealing the structure of a system. We develop a novel identifiability theorem, leveraging causal representation learning, to uncover disentangled representations of system parameters without structural assumptions. We derive a graphical criterion specifying when system parameters can be uniquely disentangled from raw trajectory data, up to permutation and diffeomorphism. Crucially, our analysis demonstrates that global causal structures provide a lower bound on the disentanglement guarantees achievable when considering local state-dependent causal structures. We instantiate system parameter identification as a variational inference problem, leveraging a sparsity-regularised transformer to uncover state-dependent causal structures. We empirically validate our approach across four synthetic domains, demonstrating its ability to recover highly disentangled representations that baselines fail to recover. Corroborating our theoretical analysis, our results confirm that enforcing local causal structure is often necessary for full identifiability.
76	5b282bb7d026d90537274a7d1db0e134cc44ae5d		10.54097/ynsrha65	2026	Multi-Scale Entropy for Transformers: Interpreting Training Dynamics and Guiding an Adaptive Training Pipeline	Jiahan Xie	1	Mathematical Modeling and Algorithm Application	0	The internal training dynamics of Transformer models remain poorly understood, limiting the development of both interpretable diagnostics and efficient training strategies. This paper proposes a unified multi-scale entropy framework that serves both as an analytical tool to interpret training dynamics and as a practical controller for adaptive optimization. It is demonstrated through a dual-track investigation. Analytically, it is used to dissect a BERT model's attention patterns on linguistic phenomena, revealing a clear layer-wise hierarchy of functional specialization. Second, the framework is operationalized to develop an entropy-guided adaptive training strategies and finally form a pipeline for Vision Transformers (ViT) which uses the stabilization of mid-layer attention entropy, a robust convergence signal identified through analysis, to trigger intelligent early stopping and staged optimization. Experiment shows the adaptive pipeline achieves performance comparable to a full training schedule while significantly reducing computational costs. Ultimately, this work establishes multi-scale entropy as a cohesive bridge between understanding how Transformers learn and controlling how they are trained, providing a versatile framework for enhancing both model interpretability and training efficiency.
77	97ed18b9f4197d71ad7a6bc4dade4f69b15cd139	2603.07162	10.48550/arXiv.2603.07162	2026	Spectral Conditioning of Attention Improves Transformer Performance	Hemanth Saratchandran	2	Neural Information Processing Systems	9	We present a theoretical analysis of the Jacobian of an attention block within a transformer, showing that it is governed by the query, key, and value projections that define the attention mechanism. Leveraging this insight, we introduce a method that systematically alters the spectral properties of each attention layer to reduce the Jacobian's condition number, thereby improving the overall conditioning of the attention layers within a transformer network. We empirically show that this improved Jacobian conditioning translates to enhanced performance in practice. Our approach is simple, broadly applicable, and can be easily integrated as a drop-in replacement for a wide range of existing attention mechanisms. We validate its effectiveness across diverse transformer architectures and tasks, demonstrating consistent improvements in performance.
78	e996e428d3eff73e936ee337f1352705420d0790	2603.06248	10.48550/arXiv.2603.06248	2026	Gradient Flow Polarizes Softmax Outputs towards Low-Entropy Solutions	A. Varre	3	arXiv.org	0	Understanding the intricate non-convex training dynamics of softmax-based models is crucial for explaining the empirical success of transformers. In this article, we analyze the gradient flow dynamics of the value-softmax model, defined as ${L}(\mathbf{V} \sigma(\mathbf{a}))$, where $\mathbf{V}$ and $\mathbf{a}$ are a learnable value matrix and attention vector, respectively. As the matrix times softmax vector parameterization constitutes the core building block of self-attention, our analysis provides direct insight into transformer's training dynamics. We reveal that gradient flow on this structure inherently drives the optimization toward solutions characterized by low-entropy outputs. We demonstrate the universality of this polarizing effect across various objectives, including logistic and square loss. Furthermore, we discuss the practical implications of these theoretical results, offering a formal mechanism for empirical phenomena such as attention sinks and massive activations.
79	bbdca14522ab6d86d730ea7865071f68cd65cae7	2603.01285	10.48550/arXiv.2603.01285	2026	Attention Smoothing Is All You Need For Unlearning	S. Z. Zade	4	arXiv.org	8	Large Language Models are prone to memorizing sensitive, copyrighted, or hazardous content, posing significant privacy and legal concerns. Retraining from scratch is computationally infeasible, whereas current unlearning methods exhibit unstable trade-offs between forgetting and utility, frequently producing incoherent outputs on forget prompts and failing to generalize due to the persistence of lexical-level and semantic-level associations in attention. We propose Attention Smoothing Unlearning (ASU), a principled framework that casts unlearning as self-distillation from a forget-teacher derived from the model's own attention. By increasing the softmax temperature, ASU flattens attention distributions and directly suppresses the lexical-level and semantic-level associations responsible for reconstructing memorized knowledge. This results in a bounded optimization objective that erases factual information yet maintains coherence in responses to forget prompts. Empirical evaluation on TOFU, MUSE, and WMDP, along with real-world and continual unlearning scenarios across question answering and text completion, demonstrates that ASU outperforms the baselines for most unlearning scenarios, delivering robust unlearning with minimal loss of model utility.
80	bc86a5468d86181370f6d0915bd639151ccf6de4		10.1016/j.eswa.2026.131812	2026	Adaptive deep learning for slide-level multi-label biomarker prediction in breast cancer whole slide images via misprediction risk analysis	Gul Sheeraz	5	Expert systems with applications	0
81	382a59f92d24b938fbca6c8b0eb4b338a9e8e800		10.1016/j.ecoinf.2026.103743	2026	Explainable AI -driven deep learning model for the root image-based classification of the medicinal plants Saposhnikovia divaricata, Glehnia littoralis, and Peucedanum japonicum	Hyein Lee	8	Ecological Informatics	0
82	6e8bc61fefd9c4dfce67b1a3784e396cb38b6c37	2602.20433	10.48550/arXiv.2602.20433	2026	Disentangling Geometry, Performance, and Training in Language Models	Atharva Kulkarni	4	arXiv.org	2	Geometric properties of Transformer weights, particularly the unembedding matrix, have been widely useful in language model interpretability research. Yet, their utility for estimating downstream performance remains unclear. In this work, we systematically investigate the relationship between model performance and the unembedding matrix geometry, particularly its effective rank. Our experiments, involving a suite of 108 OLMo-style language models trained under controlled variation, reveal several key findings. While the best-performing models often exhibit a high effective rank, this trend is not universal across tasks and training setups. Contrary to prior work, we find that low effective rank does not cause late-stage performance degradation in small models, but instead co-occurs with it; we find adversarial cases where low-rank models do not exhibit saturation. Moreover, we show that effective rank is strongly influenced by pre-training hyperparameters, such as batch size and weight decay, which in-turn affect the model's performance. Lastly, extending our analysis to other geometric metrics and final-layer representation, we find that these metrics are largely aligned, but none can reliably predict downstream performance. Overall, our findings suggest that the model's geometry, as captured by existing metrics, primarily reflects training choices rather than performance.
83	c5142546b94cf8411cb8b8a2bb3495905442d36b	2602.18849	10.48550/arXiv.2602.18849	2026	Exact Attention Sensitivity and the Geometry of Transformer Stability	S. Emadi	1	arXiv.org	7	We develop a sensitivity analysis for transformer attention in a geometry aligned with tokenwise computation. Our main result is the exact identity $\|J_\tau(u)\|_{\infty\to1}=\theta(p)/\tau$ for the Jacobian $J_\tau(u)$ of the tempered softmax $u\mapsto\mathrm{softmax}(u/\tau)$, where $\theta(p)=4\max_{S\subseteq[L]}p(S)(1-p(S))$ measures how evenly the attention distribution can be bisected rather than how concentrated it is. We combine this identity with a block-$\infty$/RMS norm under which row-stochastic attention mixing is nonexpansive. This yields a distribution-aware local Jacobian bound for multi-head attention and a sequence-length-independent Lipschitz bound on bounded input sets, with explicit dependence on width, input magnitude, temperature, and projection norms. We also identify a structural distinction between normalization placements: a pre-LN residual-sublayer Jacobian contains an additive identity term, whereas a post-LN residual-sublayer Jacobian does not. A LayerNorm projection lemma gives a sufficient condition under which the LayerNorm-only term in the post-LN expansion contracts geometrically; the condition is not tested by our experiments. Across three Pre-LN early-training runs of $774$M-parameter models, attention becomes substantially more concentrated while the median lower-bound certificate for $\theta(p)$ remains near one at every sampled layer and checkpoint. This certifies near-maximal exact sensitivity for at least half of the sampled rows within each layer. A minority of rows enters a dominant-atom regime with lower exact sensitivity, consistent with the deterministic relationship between $p_{\max}$ and $\theta(p)$.
84	591a11cd574afa84724980aaa231a897ac760a16	2602.18851	10.48550/arXiv.2602.18851	2026	Rank-Aware Spectral Bounds on Attention Logits for Stable Low-Precision Training	S. Emadi	1	arXiv.org	2	Attention scores in transformers are bilinear forms $S_{ij} = x_i^\top M x_j / \sqrt{d_h}$ whose maximum magnitude governs overflow risk in low-precision training. We derive a \emph{rank-aware concentration inequality}: when the interaction matrix $M = W^Q W^{K\top}$ has rank $r \ll d$, tail probabilities for $\max_{i,j}|S_{ij}|$ decay as $\exp(-d^{2}\alpha^{2}/(\gamma r))$ rather than $\exp(-d\alpha^{2})$, where $\gamma>1$ is a typicality parameter. For transformer attention where $r = d_h$, this yields $8$--$28\times$ tighter concentration than rank-agnostic bounds in modern architectures. We apply this result to FP8 training, deriving \emph{geometry-aware scale factors} that provide principled overflow guarantees without observing activations. The method computes per-layer scales from the spectral norm $\|W^Q W^{K\top}\|_2$ via implicit power iteration, includes a grouped query attention formulation that avoids key expansion, and remains compatible with fused attention kernels. Across GPT-2 XL to Llama-2-70B, geometry-aware scaling eliminates overflows in transient scenarios where delayed scaling fails, while achieving comparable downstream MMLU accuracy.
85	e24374b74abb58cde8abe9687ba9dd8df7de14ec		10.70862/csir.2026.0301-18	2026	Conjugational Stability and Symmetry-Invariant Transformations in Artificial Neural Networks	K. Yotov	2	Computer Science and Interdisciplinary Research Journal	1	This paper examines the properties of algebraically stabilized linear transformations in artificial neural networks (ANNs) under a change of basis and their compatibility with symmetry-invariant architectures. We investigate the conditions under which a stabilizing transformation of the form W^k=I+μD, defined over integer matrices, preserves its properties under conjugation, i.e., when passing to an equivalent form W'=B^(-1) WB in a new basis. It is shown that such transformations form a closed class with respect to conjugational equivalence and can be employed in environments with diverse geometric and symmetry requirements. Additionally, we analyze the role of symmetric actions on the input space and define conditions for the invariance of transformations with respect to specific groups of transformations. Both formal algebraic properties and the geometric and semantic interpretation of stability in this context are considered. Different types of stability are presented – algebraic, conjugational, and symmetry-based – along with the relationships between them. The results are relevant to the design of ANN architectures with predictable behavior, robustness under transformations, and compatibility with predefined symmetric structures.
86	181c5e81d4e5de9df547ba9ce78416af6b6af48f	2602.13136	10.48550/arXiv.2602.13136	2026	Order Matters in Retrosynthesis: Structure-aware Generation via Reaction-Center-Guided Discrete Flow Matching	Chenguang Wang	4	arXiv.org	3	Template-free retrosynthesis methods treat the task as black-box sequence generation, limiting learning efficiency, while semi-template approaches rely on rigid reaction libraries that constrain generalization. We address this gap with a key insight: atom ordering in neural representations matters. Building on this insight, we propose a structure-aware template-free framework that encodes the two-stage nature of chemical reactions as a positional inductive bias. By placing reaction center atoms at the sequence head, our method transforms implicit chemical knowledge into explicit positional patterns that the model can readily capture. The proposed RetroDiT backbone, a graph transformer with rotary position embeddings, exploits this ordering to prioritize chemically critical regions. Combined with discrete flow matching, our approach decouples training from sampling and enables generation in 20--50 steps versus 500 for prior diffusion methods. Our method achieves state-of-the-art performance on both USPTO-50k (61.2% top-1) and the large-scale USPTO-Full (51.3% top-1) with predicted reaction centers. With oracle centers, performance reaches 71.1% and 63.4% respectively, surpassing foundation models trained on 10 billion reactions while using orders of magnitude less data. Ablation studies further reveal that structural priors outperform brute-force scaling: a 280K-parameter model with proper ordering matches a 65M-parameter model without it.
87	81a432173f499972319120c5f2ad507ed64e1617	2602.10410	10.48550/arXiv.2602.10410	2026	LUCID: Attention with Preconditioned Representations	Sai Surya Duvvuri	4	arXiv.org	1	Softmax-based dot-product attention is a cornerstone of Transformer architectures, enabling remarkable capabilities such as in-context learning. However, as context lengths increase, a fundamental limitation of the softmax function emerges: it tends to diffuse probability mass to irrelevant tokens degrading performance in long-sequence scenarios. Furthermore, attempts to sharpen focus by lowering softmax temperature hinder learnability due to vanishing gradients. We introduce LUCID Attention, an architectural modification that applies a preconditioner to the attention probabilities. This preconditioner, derived from exponentiated key-key similarities, minimizes overlap between the keys in a Reproducing Kernel Hilbert Space, thus allowing the query to focus on important keys among large number of keys accurately with same computational complexity as standard attention. Additionally, LUCID's preconditioning-based approach to retrieval bypasses the need for low temperature and the learnability problems associated with it. We validate our approach by training ~1 billion parameter language models evaluated on up to 128K tokens. Our results demonstrate significant gains on long-context retrieval tasks, specifically retrieval tasks from BABILong, RULER, SCROLLS and LongBench. For instance, LUCID achieves up to 18% improvement in BABILong and 14% improvement in RULER multi-needle performance compared to standard attention.
88	b8a1db9a877e6bd50f9001332481ad9296a7cbb5	2602.10408		2026	Gated Normalization Removal and Scale Anchoring in Pre-Norm Transformers	Andrei Kanavalau	3		5	Normalization layers are standard in transformers, but it is not clear whether their sample-dependent computations are necessary throughout both training and inference. This work develops a gated normalization-removal approach for pre-norm transformers. The approach is implemented using TaperNorm, which starts from standard RMSNorm/LayerNorm and gradually tapers to learned sample-independent linear or affine maps. Once the gate reaches zero, per-token statistics are no longer computed in the tapered layers and the resulting maps can be folded into adjacent linear projections. The results indicate that internal normalization can be tapered in the tested pre-training and fine-tuning settings with small validation-loss increases. Our approach helps reveal a distinct role for final normalization, namely that it anchors the scale of the pre-logit representation. With this anchor present, radial changes in the last hidden state do not directly reduce the loss; when it is removed, reducing cross-entropy can be achieved by increasing logit magnitudes. A fixed-target scale loss provides an explicit alternative anchor and enables fully norm-free ablations in the tested regimes. Finally, in a KV-cached autoregressive decoding benchmark, tapering internal norms gives up to $1.14\times$ higher throughput with explicit scaling operations and up to $1.18\times$ after folding.
89	b11d52438fdb53fc0eca84a01ddeff396c5587a4		10.3390/math14040623	2026	Algebraic Stabilization of Linear Transformations in Artificial Neural Networks	K. Yotov	3	Mathematics	2	This study proposes a new formalized approach to the stabilization of linear transformations in artificial neural networks, based on discrete algebraic properties. In contrast to existing stability methods that rely on spectral norms, regularization techniques, or empirical heuristics, this work introduces the concept of algebraic stabilization—stability that arises from the structural properties of the matrices defining linear operators. The central object of investigation is the class of integer-valued matrices for which exponentiation to a form of the type Wk=I+μD is possible, where D∈Zn×n,μ∈Z>1. A well-known problem in group algebra is considered that guarantees the existence of such an exponent under the condition that μ is coprime with the determinant of W. Within this framework, modular arithmetic, reduction modulo μ, and the group structure of GLn(Zμ) are employed, thereby linking the proposed method to the theory of finite groups and linear automata. The advantages of the approach are discussed, including formal control over the iterative behavior of transformations, compatibility with quantized and finitely representable networks, the possibility of embedding stabilizing conditions directly into the network architecture, and the potential to improve model interpretability and reliability. At the same time, limitations are identified, particularly those related to constructive implementation, the selection of suitable hyperparameters, and generalization to broader classes of transformations.
90	d86decddd257acc8d1fa00a13240f3008e3afeb2	2602.08216		2026	Entropy-Generated Attention Beyond Softmax and Entmax: Kaniadakis and Reciprocal-Symmetric Abe Operators	Gunn Kim	1		2	We derive two attention operators from generalized statistical entropies. Kaniadakis entropy yields an exact full-support normalization whose weights and low-score sensitivities decay algebraically, rather than exponentially as in Softmax or by exact truncation as in entmax. Classical Abe entropy yields an implicit reciprocal-symmetric operator. With $q=e^\epsilon$, the involution $q\leftrightarrow q^{-1}$ removes every odd correction about Softmax; we obtain the normalized second- and fourth-order terms, including the deformation of the normalization multiplier. These stationary laws follow from a Fisher-metric Lagrangian on the probability simplex, whose Shannon sector recovers scaled dot-product Softmax. We also give a tangent-gradient test for deciding whether changing the entropy changes the attention profile or only its scale. R\'enyi and two-parameter Sharma--Mittal entropies retain the Tsallis--entmax inverse-gradient shape, but their global moments make the effective temperature input dependent when the external temperature is fixed. Distinguishing profile-shape equivalence from fixed-parameter operator equivalence separates new normalization shapes from adaptive rescalings and organizes the operators by support, tail behavior, and realization complexity.
91	93a2be9e6e945df049b392498ac001e7bdb4ce2a	2602.05287	10.48550/arXiv.2602.05287	2026	Position: Universal Time Series Foundation Models Rest on a Category Error	Xilin Dai	4	arXiv.org	4	This position paper argues that the pursuit of"Universal Foundation Models for Time Series"rests on a fundamental category error, mistaking a structural Container for a semantic Modality. We contend that because time series hold incompatible generative processes (e.g., finance vs. fluid dynamics), monolithic models degenerate into expensive"Generic Filters"that fail to generalize under distributional drift. To address this, we introduce the"Autoregressive Blindness Bound,"a theoretical limit proving that history-only models cannot predict intervention-driven regime shifts. We advocate replacing universality with a Causal Control Agent paradigm, where an agent leverages external context to orchestrate a hierarchy of specialized solvers, from frozen domain experts to lightweight Just-in-Time adaptors. We conclude by calling for a shift in benchmarks from"Zero-Shot Accuracy"to"Drift Adaptation Speed"to prioritize robust, control-theoretic systems.
92	687e684c1b8cb8ffb03e2b3e0cb99eb5280ae54d	2602.01308	10.48550/arXiv.2602.01308	2026	Dispelling the Curse of Singularities in Neural Network Optimizations	Heng-Jie Cao	14	arXiv.org	0	This work investigates the optimization instability of deep neural networks from a less-explored yet insightful perspective: the emergence and amplification of singularities in the parametric space. Our analysis reveals that parametric singularities inevitably grow with gradient updates and further intensify alignment with representations, leading to increased singularities in the representation space. We show that the gradient Frobenius norms are bounded by the top singular values of the weight matrices, and as training progresses, the mutually reinforcing growth of weight and representation singularities, termed the curse of singularities, relaxes these bounds, escalating the risk of sharp loss explosions. To counter this, we propose Parametric Singularity Smoothing (PSS), a lightweight, flexible, and effective method for smoothing the singular spectra of weight matrices. Extensive experiments across diverse datasets, architectures, and optimizers demonstrate that PSS mitigates instability, restores trainability even after failure, and improves both training efficiency and generalization.
93	b73390d9a806fd56c03d1749253a8ef29307cc47		10.1109/JIOT.2025.3633705	2026	ActiFormer: Sign-Aware Linear Attention for Sensor-Based Human Activity Recognition	Qifan Sun	7	IEEE Internet of Things Journal	2	Human activity recognition (HAR) plays a pivotal role in ubiquitous computing. However, it remains constrained by the challenge of balancing fine-grained temporal modeling with real-time efficiency on resource-limited devices. While Transformer-based models excel at capturing long-range dependencies, they suffer from high computational costs, limiting their applicability on resource-constrained devices. Linear attention mechanisms improve efficiency but often discard negative signals and produce overly smooth, high-entropy attention distributions, impairing the extraction of fine-grained patterns and degrading classification accuracy in complex scenarios. In this work, we present ActiFormer, a novel sign-aware linear attention framework tailored for sensor-based HAR to overcome these limitations. To preserve bidirectional signal dynamics, we introduce sign-aware attention, which explicitly models both same-sign and cross-sign interactions between queries and keys, effectively retaining negative-signals crucial for accurate recognition. Furthermore, we propose a learnable entropy-scaling function that compensates for the exponential scaling effect lost in linear attention, originally provided by softmax, solving the high-entropy attention weight issue by amplifying the importance of critical temporal points. Extensive experiments on four benchmark HAR datasets demonstrate that ActiFormer consistently outperforms CNNs, standard Transformers, and state-of-the-art (SOTA) linear attention models, both in accuracy and efficiency. Its lightweight design supports real-time inference on edge devices such as the Raspberry Pi 5, highlighting its practical deployability in real-world applications.
94	00170d59d70b5b9840e3efbb07a1e9ecf3f25f89	2601.09588	10.48550/arXiv.2601.09588	2026	Energy-Entropy Regularization: The True Power of Minimal Looped Transformers	Wai Lam	1	arXiv.org	3	Recent research suggests that looped Transformers have superior reasoning capabilities compared to standard deep architectures. Current approaches to training single-head looped architectures on benchmark tasks frequently fail or yield suboptimal performance due to a highly non-convex and irregular loss landscape. In these settings, optimization often stagnates in poor local minima and saddle points of the loss landscape, preventing the model from discovering the global minimum point. The internal mechanisms of these single-head looped transformer models remain poorly understood, and training them from scratch remains a significant challenge. In this paper, we propose a novel training framework that leverages Tsallis entropy and Hamiltonian dynamics to transform the geometry of the loss landscape. By treating the parameter updates as a physical flow, we successfully trained a single-head looped Transformer with model dimension $d = 8$ to solve induction head task with input sequence length of 1000 tokens. This success reveals the internal mechanism behind the superior reasoning capability.
95	df2bbb30d0cc3b68d8bf5ad2de173459df9c0620		10.1007/s11227-026-08225-x	2026	TFRec:a time-frequency model for capturing periodic preferences in sequential recommendation	Ke-Lei Sun	3	Journal of Supercomputing	0
96	31317d7833d556ddbd43c6426c9387cedb320f05	2512.21691	10.48550/arXiv.2512.21691	2025	Analyzing the Mechanism of Attention Collapse in VGGT from a Dynamics Perspective	Huan Li	4	arXiv.org	7	Visual Geometry Grounded Transformer (VGGT) delivers state-of-the-art feed-forward 3D reconstruction, yet its global self-attention layer suffers from a drastic collapse phenomenon when the input sequence exceeds a few hundred frames: attention matrices rapidly become near rank-one, token geometry degenerates to an almost one-dimensional subspace, and reconstruction error accumulates super-linearly.In this report,we establish a rigorous mathematical explanation of the collapse by viewing the global-attention iteration as a degenerate diffusion process.We prove that,in VGGT, the token-feature flow converges toward a Dirac-type measure at a $O(1/L)$ rate, where $L$ is the layer index, yielding a closed-form mean-field partial differential equation that precisely predicts the empirically observed rank profile.The theory quantitatively matches the attention-heat-map evolution and a series of experiments outcomes reported in relevant works and explains why its token-merging remedy -- which periodically removes redundant tokens -- slows the effective diffusion coefficient and thereby delays collapse without additional training.We believe the analysis provides a principled lens for interpreting future scalable 3D-vision transformers,and we highlight its potential for multi-modal generalization.
97	2393160c40396112c23fdc9cf5fa7a39fecdc649		10.1016/j.eswa.2025.130830	2025	GRPO-TST: Group relative policy optimized time series tokenization for long-horizon equipment-state forecasting in continuous casting	Qi Zhang	5	Expert systems with applications	0
98	7f5d2907a0f2f9914ad5adae4c28321f3d6b4aca		10.1109/TITS.2025.3622979	2025	Position-Invariant Graph Convolutional Recurrent Network for Traffic Forecasting	Xiao Liu	8	IEEE transactions on intelligent transportation systems (Print)	0	Traffic forecasting leverages multivariate time series analysis to predict traffic patterns. Real-world traffic data comprises two distinct types of latent time-series signals: diffuse signals, which refer to time-varying information propagated across the traffic network, and intrinsic signals, which capture unique, location-specific patterns. However, existing approaches often treat traffic signals solely as diffusion outcomes, overlooking the intrinsic characteristics that can significantly influence model performance. To address this issue, we propose the Position-invariant Graph Convolutional Recurrent Network (PGCRN), which decouples diffuse and intrinsic signals for improved traffic forecasting. Instead of relying on a predefined graph, PGCRN learns graph structures from spatio-temporal data through a learnable position-invariant node representation that forms an adaptive adjacency matrix. This is integrated into a Graph Convolutional Recurrent Network (GCRN) encoder–decoder to jointly capture spatial and temporal dependencies. Furthermore, we introduce a contrastive learning framework in which a node’s time-varying and position-invariant representations form positive pairs, while position-invariant representations from different nodes form negative pairs. The model is trained with a triplet loss. Experiments on four benchmark datasets show that PGCRN consistently outperforms strong baselines. Owing to its computational efficiency, PGCRN is also well suited for deployment on resource-constrained edge devices.
99	652b1768e9627342a1d696d05cf54c08a32afdde	2511.21377	10.48550/arXiv.2511.21377	2025	Controlling changes to attention logits	Ben Anson	2	arXiv.org	3	Stability of neural network weights is critical when training transformer models. The query and key weights are particularly problematic, as they tend to grow large without any intervention. Applying normalization to queries and keys, known as `QK norm', fixes stability issues in practice, but is not always applicable. For example, QK norm is not compatible with Multi Latent Attention (MLA) because QK norm requires full materialization of queries and keys during inference, which is not done in MLA. In this paper we suggest that controlling the changes to logits is important for stability. We show that these changes are controllable by assigning parameter-dependent learning rates to the query and key weights. We find that our cheap intervention allows us to increase the base learning rate of the network, outperform other methods in the MLA setting, and achieve performance competitive with QK norm when using Multi-head Attention.
100	8d7719f045a5030df5c0e5af179379debf775793		10.1109/FLLM67465.2025.11390929	2025	AI Feels Time: Augmenting LLMs with a Bio-Inspired Temporal Cognitive Notebook : FLLM2025	Naba Alghudran	1	2025 3rd International Conference on Foundation and Large Language Models (FLLM)	0	Large Language Models (LLMs) exhibit remarkable linguistic capabilities but often struggle with complex, multi-step reasoning tasks. We hypothesize that this limitation stems, in part, from the absence of temporal cognition—the capacity to represent, track, and reason over time. Inspired by neuroscience, particularly the role of "time cells" in the hippocampus and entorhinal cortex, we introduce the Temporal Cognitive Notebook (TCN): a novel architecture that integrates adaptive computation, temporally structured memory, and entropy-based metacognitive control. The TCN enables LLMs to dynamically allocate reasoning resources and monitor progress over time, promoting more patient and context-aware reasoning. Theoretical analyses and preliminary evaluations suggest that embedding temporal cognition could bridge a fundamental gap between human and artificial reasoning.
101	ae99cab5756ddf92bd39000dc01b9eb0de5a1856	2511.20698	10.48550/arXiv.2511.20698	2025	On the Role of Hidden States of Modern Hopfield Network in Transformer	Tsubasa Masumura	2	Neural Information Processing Systems	2	Associative memory models based on Hopfield networks and self-attention based on key-value mechanisms have been popular approaches in the study of memory mechanisms in deep learning. It has been pointed out that the state update rule of the modern Hopfield network (MHN) in the adiabatic approximation is in agreement with the self-attention layer of Transformer. In this paper, we go beyond this approximation and investigate the relationship between MHN and self-attention. Our results show that the correspondence between Hopfield networks and Transformers can be established in a more generalized form by adding a new variable, the hidden state derived from the MHN, to self-attention. This new attention mechanism, modern Hopfield attention (MHA), allows the inheritance of attention scores from the input layer of the Transformer to the output layer, which greatly improves the nature of attention weights. In particular, we show both theoretically and empirically that MHA hidden states significantly improve serious problem of deep Transformers known as rank collapse and token uniformity. We also confirm that MHA can systematically improve accuracy without adding training parameters to the Vision Transformer or GPT. Our results provide a new case in which Hopfield networks can be a useful perspective for improving the Transformer architecture.
102	702687f80a0586227491744567bc91783b02fc7d		10.1109/SmartIoT66867.2025.00060	2025	Entropy Regularized Transformer Autoencoder For Anomaly Detection in Multivariate Time Series	Mostofa Fahim Hasan	5	International Conferences on Smart Internet of Things	0	Anomaly detection in multivariate time series data plays a critical role in the early detection of system failure, multifunction, or cyber attack. Transformer Autoencoder (TAE) is now used in understanding behaviors across multiple inter-connected variables over time in a complex system by measuring the uncertainty in the latent parameter. Traditional autoencoders (AE) are trained to maximize the anomaly score to identify the anomaly based on the reconstruction error across all inputs, while treating all input features equally. In high-dimensional large-scale data, individual features contributed unequally to the reconstruction error and created uncertainty in the reconstruction, and we need to account for it in the anomaly detection algorithm. To focus on this issue, in this paper, we have proposed an entropy regularized transformer autoencoder (ETAE) in large-scale multivariate time-series data to quantify the uncertainty in anomaly detection. The proposed ETAE incorporates entropy-based regularization into its attention mechanism. The proposed ETAE improves the model’s anomaly detection skills by penalizing high-entropy attention distributions and encouraging the machine to pay more confidently to key regions. A thorough mathematical expression of the entropy-regularized loss has been derived, and the theoretical derivation has been verified through extensive simulation work on different multivariate time-series datasets, such as CAN car hacking, ToN_IoT network, and weather. The simulation results show that our model performs noticeably better than the conventional autoencoder and TAE baselines in terms of F1-score, accuracy, precision, and recall.
103	07fbbca748c3486f72155bcfa04da6575cf3ccb6	2511.11553	10.1109/TAC.2026.3687461	2025	Multistability of Self-Attention Dynamics in Transformers	C. Altafini	1	IEEE Transactions on Automatic Control	10	In machine learning, a self-attention dynamics is a continuous-time multiagent-like model of the attention mechanisms of transformers. In this article, we show that such dynamics is related to a multiagent version of the Oja flow, a dynamical system that computes the principal eigenvector of a matrix corresponding for transformers to the value matrix. We classify the equilibria of the “single-head” self-attention system into four classes: consensus, bipartite consensus, clustering, and polygonal equilibria. Multiple asymptotically stable equilibria from the first three classes often coexist in the self-attention dynamics. Interestingly, equilibria from the first two classes are always aligned with the eigenvectors of the value matrix, often but not exclusively with the principal eigenvector.
104	fac4919b13bda208e555a3ae23ca212ffc49a71b	2511.03190		2025	Efficient Linear Attention for Multivariate Time Series Modeling via Entropy Calibration	Ming-Tao Zhang	5		0	Attention mechanisms are effective for multivariate time series modeling because they adaptively aggregate information across timestamps and variables. Their quadratic cost, however, limits their use in multivariate and spatio-temporal systems. Linear attention reduces this cost by replacing nonlinear attention mappings with forms that support associative computation. Replacing exponential weighting may weaken the amplification of high-score positions and further bias weakly selective distributions toward uniform aggregation. We propose an entropy-calibrated linear attention module and instantiate it as the ELinFormer forecaster. Our operator decomposes aggregation into a uniform component and a query-dependent correction, whose magnitude is calibrated by a query-specific temperature. We derive a second-order KL proxy from centered query-key scores. The Taylor expansion reduces the proxy to the squared-score energy divided by the squared temperature. This score energy can be computed from shared products of the centered keys without forming the full pairwise score matrix, and the same products support the linear attention output. We design a smooth temperature rule to calibrate the KL proxy: it keeps low-energy residuals vanishing, approximately calibrates moderate-energy queries to a shared KL level, and allows the proxy to continue growing for high-energy queries. Experiments on five PEMS benchmarks show competitive forecasting accuracy. Controlled comparisons with vanilla attention and Linformer demonstrate a favorable accuracy--memory trade-off, while analysis on PEMS08 shows that the proposed rational temperature rule and query-specific calibration achieve lower MAE than a linear rule or shared-temperature alternatives. These results support entropy calibration as an effective mechanism for preserving selective deviations in linear attention.
105	7c0996af92daf5b0471a72a11f89737407043930		10.36001/phmconf.2025.v17i1.4301	2025	Enhancing Nuclear Safeguards with Time Series Sketching-Based Nuclear Material Loss Tracking	Hao Huang	4	Annual Conference of the PHM Society	0	Detecting nuclear material loss or leak events is a critical challenge for nuclear safeguards and material accountancy in the recycling of used nuclear fuel. The MAYER (Multi-Sensor Assimilation Yielding Enhanced Reliability) project, part of the ARPA-E CURIE (Converting UNF Radioisotopes Into Energy) program, aims to develop a framework for tracking material loss in nuclear facilities by integrating multisensor data with predictive modeling. In this paper, we introduce MAterial Loss Tracking via Time Series Sketching (MALTS), a deep learning-based method designed to detect material loss events across the nuclear fuel recycling system. MALTS enhances the accuracy and robustness of nuclear material loss tracking by employing time series sketching to capture essential patterns while filtering out sensor noise, resulting in more stable predictions despite sensor noise effects. This approach also improves the time efficiency of material loss tracking by reducing the dimensionality of highfrequency sensor data, thereby enhancing computational scalability and enabling real-time inference. To further provide insights into the leak, MALTS ranks anomalous channels by post-processing results with a pretrained vision-language model (VLM) that considers the system flow diagram, generating a sorted list of anomalous channels from upstream to downstream. The initial leak location is identified as the first upstream channel. Experimental results demonstrate MALTS’s effectiveness and efficiency in accurately identifying unseen nuclear material loss events and pinpointing initial leak locations, making it suitable for deployment within the MAYER digital twin framework for nuclear material safeguards.
106	80d51d8ee4f4c7c32e3cb101e77d4050e0d97608	2510.14254	10.48550/arXiv.2510.14254	2025	Generalist vs Specialist Time Series Foundation Models: Investigating Potential Emergent Behaviors in Assessing Human Health Using PPG Signals	Saurabh Kataria	24	arXiv.org	4	Foundation models are large-scale machine learning models that are pre-trained on massive amounts of data and can be adapted for various downstream tasks. They have been extensively applied to tasks in Natural Language Processing and Computer Vision with models such as GPT, BERT, and CLIP. They are now also increasingly gaining attention in time-series analysis, particularly for physiological sensing. However, most time series foundation models are specialist models - with data in pre-training and testing of the same type, such as Electrocardiogram, Electroencephalogram, and Photoplethysmogram (PPG). Recent works, such as MOMENT, train a generalist time series foundation model with data from multiple domains, such as weather, traffic, and electricity. This paper aims to conduct a comprehensive benchmarking study to compare the performance of generalist and specialist models, with a focus on PPG signals. Through an extensive suite of total 51 tasks covering cardiac state assessment, laboratory value estimation, and cross-modal inference, we comprehensively evaluate both models across seven dimensions, including win score, average performance, feature quality, tuning gain, performance variance, transferability, and scalability. These metrics jointly capture not only the models'capability but also their adaptability, robustness, and efficiency under different fine-tuning strategies, providing a holistic understanding of their strengths and limitations for diverse downstream scenarios. In a full-tuning scenario, we demonstrate that the specialist model achieves a 27% higher win score. Finally, we provide further analysis on generalization, fairness, attention visualizations, and the importance of training data choice.
107	0a849c8a9c28837a80dce67ef114e22f7110c20f	2510.09017	10.48550/arXiv.2510.09017	2025	Value-State Gated Attention for Mitigating Extreme-Token Phenomena in Transformers	Rui Bu	4	arXiv.org	8	Large models based on the Transformer architecture are susceptible to extreme-token phenomena, such as attention sinks and value-state drains. These issues, which degrade model performance, quantization fidelity, and interpretability, arise from a problematic mutual reinforcement mechanism where the model learns an inefficient'no-op'behavior by focusing attention on tokens with near-zero value states. In this paper, we propose Value-State Gated Attention (VGA), a simple, dedicated, and stable architectural mechanism for performing'no-op'attention efficiently by directly breaking this cycle. VGA introduces a learnable, data-dependent gate, computed directly from the value vectors (V), to modulate the output. Through a theoretical analysis of the underlying gradients, we show that gating the value-state with a function of itself is more effective at decoupling value and attention score updates than prior methods that gate on input embeddings. This creates a direct regulatory pathway that allows the model to suppress a token's contribution based on its emergent value representation. Our experiments demonstrate that VGA significantly mitigates the formation of attention sinks and stabilizes value-state norms, leading to improved performance, robust quantization fidelity, and enhanced model interpretability.
108	bf7171b89dfae23c357db155343292b525f28cb1	2510.13832	10.48550/arXiv.2510.13832	2025	Entropy Meets Importance: A Unified Head Importance-Entropy Score for Stable and Efficient Transformer Pruning	Minsik Choi	4	arXiv.org	2	Transformer-based models have achieved remarkable performance in NLP tasks. However, their structural characteristics-multiple layers and attention heads-introduce efficiency challenges in inference and deployment. To address these challenges, various pruning methods have recently been proposed. Notably, gradient-based methods using Head Importance Scores (HIS) have gained traction for interpretability, efficiency, and ability to identify redundant heads. However, HIS alone has limitations as it captures only the gradient-driven contribution, overlooking the diversity of attention patterns. To overcome these limitations, we introduce a novel pruning criterion, HIES (Head Importance-Entropy Score), which integrates head importance scores with attention entropy, providing complementary evidence on per-head contribution. Empirically, HIES-based pruning yields up to 15.2% improvement in model quality and 2.04x improvement in stability over HIS-only methods, enabling substantial model compression without sacrificing either accuracy or stability. Code will be released upon publication.
109	411bd0430d1761139e29765f02f0859e5473d4da	2510.00517	10.48550/arXiv.2510.00517	2025	Understanding Sensitivity of Differential Attention through the Lens of Adversarial Robustness	Tsubasa Takahashi	4	arXiv.org	0	Differential Attention (DA) has been proposed as a refinement to standard attention, suppressing redundant or noisy context through a subtractive structure and thereby reducing contextual hallucination. While this design sharpens task-relevant focus, we show that it also introduces a structural fragility under adversarial perturbations. Our theoretical analysis identifies negative gradient alignment-a configuration encouraged by DA's subtraction-as the key driver of sensitivity amplification, leading to increased gradient norms and elevated local Lipschitz constants. We empirically validate this Fragile Principle through systematic experiments on ViT/DiffViT and evaluations of pretrained CLIP/DiffCLIP, spanning five datasets in total. These results demonstrate higher attack success rates, frequent gradient opposition, and stronger local sensitivity compared to standard attention. Furthermore, depth-dependent experiments reveal a robustness crossover: stacking DA layers attenuates small perturbations via depth-dependent noise cancellation, though this protection fades under larger attack budgets. Overall, our findings uncover a fundamental trade-off: DA improves discriminative focus on clean inputs but increases adversarial vulnerability, underscoring the need to jointly design for selectivity and robustness in future attention mechanisms.
110	919db1934f9fb93700e5c0c8d47a048a2c3c466e	2509.13907	10.48550/arXiv.2509.13907	2025	White Aggregation and Restoration for Few-shot 3D Point Cloud Semantic Segmentation	J. Im	4	arXiv.org	0	Few-shot 3D Point Cloud Semantic Segmentation (FS-PCS) aims to predict per-point labels for an unlabeled point cloud, given only a few labeled examples. To extract representations from the limited labeled set, existing methods have constructed prototypes with Farthest Point Sampling (FPS). However, we found that this convention results in performance instability due to its sensitivity to FPS-induced variations, while the prototype generation process remains underexplored in the field. This motivates us to investigate deterministic prototype generation method based on attention mechanism. Despite its potential, we found that vanilla attention module suffers from the distributional gap between prototypical tokens and support features. To overcome this, we provide a simple approach, White Aggregation and Restoration Module (WARM), which resolves the misalignment by wrapping cross-attention with whitening and coloring transformations. Specifically, whitening aligns the features to tokens before the attention process, and coloring subsequently restores the original distribution to the attended tokens. This design enables robust attention, thereby generating prototypes that capture the semantic relationships in support features. WARM achieves state-of-the-art performance with a significant margin on the S3DIS dataset, and competitive performance on the ScanNet dataset. Further experiments demonstrate its effectiveness in deterministic prototype generation. Code is publicly available at: https://github.com/JiyunIm00/WARM.git
111	de13268a28955d312fce2b1eefe77a4f5e4aa372	2509.07324	10.48550/arXiv.2509.07324	2025	Mitigating Attention Localization in Small Scale: Self-Attention Refinement via One-step Belief Propagation	Nakyung Lee	7	Conference on Empirical Methods in Natural Language Processing	4	Transformer-based self-attention mechanism serves as the core of modern language models, yet it often suffers from localization, where attentions collapse onto a limited subset of tokens and fail to capture long-range dependencies. To address this issue, we propose Self-Attention One-step Belief Propagation (SAOBP), a refinement framework that injects multi-hop relationships through a belief propagation process. To interpret and quantify these interactions, we introduce Global Token Dependency (GTD) that captures the relative contribution of multihop connections within the attention graph. Empirical results indicate that SAOBP helps prevent entropy collapse in deeper layers and adaptively maintains GTD at task-appropriate levels, thereby supporting improvements in model performance. Importantly, we observe competitive gains in small-scale models, highlighting its potential for improving inference quality in resource-constrained scenarios.
112	b36a2d1d202d6786d9d6cf9add5368db3e0a4cf8		10.1016/j.eswa.2025.129649	2025	GE-adapter: A general and efficient adapter for enhanced video editing with pretrained text-to-image diffusion models	Yang-Fan He	11	Expert systems with applications	17
113	d40e7849406064670e0cf92160221f8490925844	2509.10501	10.48550/arXiv.2509.10501	2025	From Noise to Precision: A Diffusion-Driven Approach to Zero-Inflated Precipitation Prediction	Wen-Tao Gao	9	European Conference on Artificial Intelligence	2	Zero-inflated data pose significant challenges in precipitation forecasting due to the predominance of zeros with sparse non-zero events. To address this, we propose the Zero Inflation Diffusion Framework (ZIDF), which integrates Gaussian perturbation for smoothing zero-inflated distributions, Transformer-based prediction for capturing temporal patterns, and diffusion-based denoising to restore the original data structure. In our experiments, we use observational precipitation data collected from South Australia along with synthetically generated zero-inflated data. Results show that ZIDF demonstrates significant performance improvements over multiple state-of-the-art precipitation forecasting models, achieving up to 56.7\% reduction in MSE and 21.1\% reduction in MAE relative to the baseline Non-stationary Transformer. These findings highlight ZIDF's ability to robustly handle sparse time series data and suggest its potential generalizability to other domains where zero inflation is a key challenge.
114	71434d21bf0e4e939c7448618379de042b357b41	2508.20577	10.48550/arXiv.2508.20577	2025	MERIT: Maximum-normalized Element-wise Ratio for Language Model Large-batch Training	Yang Luo	6	International Conference on Machine Learning	0	Large-batch training has become a cornerstone in accelerating the training of deep neural networks, yet it poses challenges in optimization and generalization. Existing optimizers like AdamW present performance degradation during language models'large-batch training, due to the information bottleneck in attention layers caused by the sharp increase of max attention logit. While the LAMB optimizer partially addresses this issue, some attention layers still face this issue. The reason is that $l_2$-norm-based trust ratios in LAMB are less effective in directly influencing the max value of query/key weights. Furthermore, the weight-wise trust ratio in LAMB is error-prone as it overlooks relationships of weight values within rows or columns. Building on these observations, we propose a novel optimizer, MERIT, which leverages the max-norm to calculate the trust ratio to constrain the max attention logit more effectively. Moreover, we further construct element-wise trust ratios to provide more robust update scaling by focusing on local weight structures. Extensive experiments of large-batch training across various sizes of GPT-2 models demonstrate the superior performance of MERIT. Notably, during the training of GPT-2 Medium, MERIT enables a 6k batch size without any performance degradation compared to the standard batch size (480) with 48B training tokens. This work highlights the importance of considering the max attention logit and finer-granularity trust ratio in large-batch training. It successfully improves the training stability and paves the way for larger batch usage, enabling faster development and iteration of large language models. Code is available at https://github.com/NUS-HPC-AI-Lab/MERIT.
115	43c69f8f585846cc0a19cd305dac31e648d6d391	2508.18387	10.48550/arXiv.2508.18387	2025	Integral Transformer: Denoising Attention, Not Too Much Not Too Little	I. Kobyzev	4	Conference on Empirical Methods in Natural Language Processing	6	Softmax self-attention often assigns disproportionate weight to semantically uninformative tokens such as special tokens and punctuation, a phenomenon known as attention noise. While recent methods like Cog Attention and the Differential Transformer have addressed this by introducing negative attention scores, they risk discarding useful information. In this paper, we propose the Integral Transformer, a novel self-attention mechanism that denoises attention by integrating signals sampled from the logit distribution. Our approach mitigates noise while preserving the contributions of special tokens critical for model performance. Extensive experiments demonstrate that our model outperforms vanilla, Cog, and Differential attention variants on well-established knowledge and reasoning language benchmarks. Moreover, our analysis reveals that employing vanilla self-attention in the lower Transformer layers enhances performance and that the Integral Transformer effectively balances attention distributions and reduces rank collapse in upper layers.
116	6f5242d7fa881678f0eea65cfdf235b8ba1fa1a1	2508.12587	10.48550/arXiv.2508.12587	2025	Multimodal Chain of Continuous Thought for Latent-Space Reasoning in Vision-Language Models	Tan-Hanh Pham	2	arXiv.org	25	Many reasoning techniques for large multimodal models adapt language model approaches, such as Chain-of-Thought (CoT) prompting, which express reasoning as word sequences. While effective for text, these methods are suboptimal for multimodal contexts, struggling to align audio, visual, and textual information dynamically. To explore an alternative paradigm, we propose the Multimodal Chain of Continuous Thought (MCOUT), which enables reasoning directly in a joint latent space rather than in natural language. In MCOUT, the reasoning state is represented as a continuous hidden vector, iteratively refined and aligned with visual and textual embeddings, inspired by human reflective cognition. We develop two variants: MCOUT-Base, which reuses the language model`s last hidden state as the continuous thought for iterative reasoning, and MCOUT-Multi, which integrates multimodal latent attention to strengthen cross-modal alignment between visual and textual features. Experiments on benchmarks including MMMU, ScienceQA, and MMStar show that MCOUT consistently improves multimodal reasoning, yielding up to 8.23% accuracy gains over strong baselines and improving BLEU scores up to 8.27% across multiple-choice and open-ended tasks. These findings highlight latent continuous reasoning as a promising direction for advancing LMMs beyond language-bound CoT, offering a scalable framework for human-like reflective multimodal inference. Code is available at https://github.com/Hanhpt23/OmniMod.
117	a3a1c6e3d38c2a7846a1bfee08644923be3713ff	2508.04925	10.1145/3744916.3773118	2025	Why Attention Fails: A Taxonomy of Faults in Attention-Based Neural Networks	Sigma Jahan	4	Proceedings of the 2026 IEEE/ACM 48th International Conference on Software Engineering	4	Attention mechanisms are at the core of modern neural architectures, powering systems ranging from ChatGPT to autonomous vehicles, and driving a major economic impact. However, high-profile failures, such as ChatGPT’s nonsensical outputs or Google’s suspension of Gemini’s image generation due to attention weight errors, highlight a critical gap: existing deep learning fault taxonomies might not adequately capture the unique failures introduced by attention mechanisms. This gap leaves practitioners without actionable diagnostic guidance. To address this gap, we present the first comprehensive empirical study of faults in attention-based neural networks (ABNNs). Our work is based on a systematic analysis of 555 real-world faults collected from 96 projects across ten frameworks, including GitHub, Hugging Face, and Stack Overflow. Through our analysis, we develop a novel taxonomy comprising seven attention-specific fault categories, not captured by existing work. Our results show that over half of the ABNN faults arise from mechanisms unique to attention architectures. We further analyze the root causes and manifestations of these faults through various symptoms. Finally, by analyzing symptom–root cause associations, we identify four evidence-based diagnostic heuristics that explain 33.0% of attention-specific faults and provide systematic, taxonomy-driven guidance for diagnosing faults in ABNNs.
118	4c04624c0a0c0c5c45a67a85829db08818ac03bb	2508.00264	10.48550/arXiv.2508.00264	2025	Calibrated Language Models and How to Find Them with Label Smoothing	Jerry M. Huang	3	International Conference on Machine Learning	8	Recent advances in natural language processing (NLP) have opened up greater opportunities to enable fine-tuned large language models (LLMs) to behave as more powerful interactive agents through improved instruction-following ability. However, understanding how this impacts confidence calibration for reliable model output has not been researched in full. In this work, we examine various open-sourced LLMs, identifying significant calibration degradation after instruction tuning in each. Seeking a practical solution, we look towards label smoothing, which has been shown as an effective method to regularize for overconfident predictions but has yet to be widely adopted in the supervised fine-tuning (SFT) of LLMs. We first provide insight as to why label smoothing is sufficient to maintain calibration throughout the SFT process. However, settings remain where the effectiveness of smoothing is severely diminished, in particular the case of large vocabulary LLMs (LV-LLMs). We posit the cause to stem from the ability to become over-confident, which has a direct relationship with the hidden size and vocabulary size, and justify this theoretically and experimentally. Finally, we address an outstanding issue regarding the memory footprint of the cross-entropy loss computation in the label smoothed loss setting, designing a customized kernel to dramatically reduce memory consumption without sacrificing speed or performance in comparison to existing solutions for non-smoothed losses.
119	c5de6b78d5e7b668e4b456f1ecbca718c3c35649	2507.23632	10.48550/arXiv.2507.23632	2025	On the Expressiveness of Softmax Attention: A Recurrent Neural Network Perspective	Gabriel Mongaras	2	Trans. Mach. Learn. Res.	7	Since its introduction, softmax attention has become the backbone of modern transformer architectures due to its expressiveness and scalability across a wide range of tasks. However, the main drawback of softmax attention is the quadratic memory requirement and computational complexity with respect to the sequence length. By replacing the softmax nonlinearity, linear attention and similar methods have been introduced to avoid the quadratic bottleneck of softmax attention. Despite these linear forms of attention being derived from the original softmax formulation, they typically lag in terms of downstream accuracy. While strong intuition of the softmax nonlinearity on the query and key inner product suggests that it has desirable properties compared to other nonlinearities, the question of why this discrepancy exists still remains unanswered. This work demonstrates that linear attention is an approximation of softmax attention by deriving the recurrent form of softmax attention. Using this form, each part of softmax attention can be described in the language of recurrent neural networks (RNNs). Describing softmax attention as an RNN allows for the ablation of the components of softmax attention to understand the importance of each part and how they interact. In this way, our work helps explain why softmax attention is more expressive than its counterparts.
120	b6cebc058440a7a72bfe9c5130a7bb51efa07fac	2508.00127	10.48550/arXiv.2508.00127	2025	Structured Transformations for Stable and Interpretable Neural Computation	Saleh Nikooroo	2	arXiv.org	2	Despite their impressive performance, contemporary neural networks often lack structural safeguards that promote stable learning and interpretable behavior. In this work, we introduce a reformulation of layer-level transformations that departs from the standard unconstrained affine paradigm. Each transformation is decomposed into a structured linear operator and a residual corrective component, enabling more disciplined signal propagation and improved training dynamics. Our formulation encourages internal consistency and supports stable information flow across depth, while remaining fully compatible with standard learning objectives and backpropagation. Through a series of synthetic and real-world experiments, we demonstrate that models constructed with these structured transformations exhibit improved gradient conditioning, reduced sensitivity to perturbations, and layer-wise robustness. We further show that these benefits persist across architectural scales and training regimes. This study serves as a foundation for a more principled class of neural architectures that prioritize stability and transparency-offering new tools for reasoning about learning behavior without sacrificing expressive power.
121	a80e1d0ef01a17b565e86ecae4e51cfa22d7fc96	2507.17501	10.48550/arXiv.2507.17501	2025	DNT: a Deeply Normalized Transformer that can be trained by Momentum SGD	Xianbiao Qi	7	arXiv.org	1	Transformers have become the de facto backbone of modern deep learning, yet their training typically demands an advanced optimizer with adaptive learning rate like AdamW, rather than a momentum SGDW (mSGDW). Previous works show that it is mainly due to a heavy-tailed distribution of the gradients. In this paper, we introduce a Deeply Normalized Transformer (DNT), which is meticulously engineered to overcome this limitation enabling seamless training with vanilla mSGDW while yielding comparable performance to the Transformers trained via AdamW. To be specific, in DNT, we strategically integrate normalization techniques at proper positions in the Transformers to effectively modulate the Jacobian matrices of each layer, balance the influence of weights, activations, and their interactions, and thus enable the distributions of gradients concentrated. We provide both theoretical justifications of the normalization technique used in our DNT and extensive empirical evaluation on two popular Transformer architectures to validate that: a) DNT outperforms its counterparts (\ie, ViT and GPT), and b) DNT can be effectively trained with vanilla mSGDW.
122	2b43876e3601f48c3471eb94c0502912dc25c43e	2507.07814	10.48550/arXiv.2507.07814	2025	Pay Attention to Attention Distribution: A New Local Lipschitz Bound for Transformers	Nikolay Yudin	4	arXiv.org	10	We introduce a novel upper bound on the local Lipschitz constant of the dot-product self-attention block showing its dependence on the attention map distributions. The proposed bound is not only tighter than the prior art, but for the first time, reveals how the distribution of attention probabilities shapes the local Lipschitz constant of the self-attention block. The theoretical basis of the proposed upper bound lies in the refined closed-form upper bounds on singular values of the Jacobian of softmax function. Leveraging these theoretical insights, we introduce JaSMin (Jacobian Softmax norm Minimization), a lightweight regularizer that directly controls the local Lipschitz constant of each block and, consequently, the entire model. Additionally, we discuss how the nature of the attention map distribution contributes to the gradient dynamics and, consequently, transformer training stability.
123	163bd2778291adc02e990d6cec0e7af646e82fd1	2507.03300	10.48550/arXiv.2507.03300	2025	LRM-1B: Towards Large Routing Model	Han Li	4	arXiv.org	1	Vehicle routing problems (VRPs) are central to combinatorial optimization with significant practical implications. Recent advancements in neural combinatorial optimization (NCO) have demonstrated promising results by leveraging neural networks to solve VRPs, yet the exploration of model scaling within this domain remains underexplored. Inspired by the success of model scaling in large language models (LLMs), this study introduces a Large Routing Model with 1 billion parameters (LRM-1B), designed to address diverse VRP scenarios. We present a comprehensive evaluation of LRM-1B across multiple problem variants, distributions, and sizes, establishing state-of-the-art results. Our findings reveal that LRM-1B not only adapts to different VRP challenges but also showcases superior performance, outperforming existing models. Additionally, we explore the scaling behavior of neural routing models from 1M to 1B parameters. Our analysis confirms power-law between multiple model factors and performance, offering critical insights into the optimal configurations for foundation neural routing solvers.
124	e5d15fbd78d2efb6a00b8cd8726b5b9207157e55	2507.03031	10.48550/arXiv.2507.03031	2025	On the Mathematical Impossibility of Safe Universal Approximators	Jasper Yao	1	arXiv.org	0	We establish fundamental mathematical limits on universal approximation theorem (UAT) system alignment by proving that catastrophic failures are an inescapable feature of any useful computational system. Our central thesis is that for any universal approximator, the expressive power required for useful computation is inextricably linked to a dense set of instabilities that make perfect, reliable control a mathematical impossibility. We prove this through a three-level argument that leaves no escape routes for any class of universal approximator architecture. i) Combinatorial Necessity: For the vast majority of practical universal approximators (e.g., those using ReLU activations), we prove that the density of catastrophic failure points is directly proportional to the network's expressive power. ii) Topological Necessity: For any theoretical universal approximator, we use singularity theory to prove that the ability to approximate generic functions requires the ability to implement the dense, catastrophic singularities that characterize them. iii) Empirical Necessity: We prove that the universal existence of adversarial examples is empirical evidence that real-world tasks are themselves catastrophic, forcing any successful model to learn and replicate these instabilities. These results, combined with a quantitative"Impossibility Sandwich"showing that the minimum complexity for usefulness exceeds the maximum complexity for safety, demonstrate that perfect alignment is not an engineering challenge but a mathematical impossibility. This foundational result reframes UAT safety from a problem of"how to achieve perfect control"to one of"how to operate safely in the presence of irreducible uncontrollability,"with profound implications for the future of UAT development and governance.
125	e33cf4875ef27c370b76a31aa9307c04d83c7ef4	2507.00754		2025	Enhancing Self-Supervised Visual Representation Learning via Low-Rank Adapted LLMs	S. Kuzucu	5		1	The integration of Large Language Model (LLM) blocks with Vision Transformers (ViTs) holds significant promise for vision-only tasks by leveraging the rich semantic knowledge and reasoning capabilities of LLMs. However, a fundamental challenge lies in the inherent modality mismatch between the text-centric pre-training of LLMs and the vision-centric training of ViTs. Direct fusion often fails to fully exploit the LLM's potential and suffers from unstable finetuning. Consequently, prior works typically keep LLM blocks frozen while learning only the vision components. To address these challenges, we introduce Language-Adapted Vision Enhancer (LAVIE), a novel framework that bridges this modality gap through a synergistic pre-training strategy. LAVIE co-adapts a ViT backbone and an LLM fusion block by (1) employing Masked Auto-Encoding (MAE) to pre-train the ViT for richer visual representations, and (2) concurrently training Low-Rank Adaptation (LoRA) layers within the LLM block using the same MAE objective. This joint optimization guides the ViT to produce LLM-aligned features and the LLM to effectively interpret visual information. We demonstrate through extensive experiments that LAVIE significantly improves performance in various downstream vision tasks, offering an effective and efficient way to enhance visual understanding using frozen LLM knowledge. Code is available at https://github.com/selimkuzucu/LAVIE
126	1ba8d8a943b8a9dea893260399dbcbd5de6166dc	2506.16640	10.48550/arXiv.2506.16640	2025	Long-Context Generalization with Sparse Attention	Pavlo Vasylenko	4	arXiv.org	23	Transformer-based architectures traditionally employ softmax to compute attention weights, which produces dense distributions over all tokens in a sequence. While effective in many settings, this density has been shown to be detrimental for tasks that demand precise focus on fixed-size patterns: as sequence length increases, non-informative tokens accumulate attention probability mass, leading to dispersion and representational collapse. We show in this paper that dynamically sparse attention mechanisms using $\alpha$-entmax can avoid these issues, due to their ability to assign exact zeros to irrelevant tokens. Furthermore, we introduce Adaptive-Scalable Entmax (ASEntmax), which endows $\alpha$-entmax with a learnable temperature parameter, allowing the attention distribution to interpolate between sparse (pattern-focused) and dense (softmax-like) regimes. Our empirical evaluation on synthetic tasks and language modeling demonstrates that ASEntmax substantially outperforms softmax, scalable softmax, and fixed-temperature $\alpha$-entmax baselines, achieving up to 1000$\times$ length extrapolation on synthetic benchmarks and superior long-context generalization on language modeling while preserving short-context performance, including better perplexity trends and higher retrieval accuracies at 8$\times$ training length.
127	07dda22342a74b911b4a5dc2cff450640db4cf38	2506.12903	10.48550/arXiv.2506.12903	2025	Variational Learning Finds Flatter Solutions at the Edge of Stability	Avrajit Ghosh	8	Neural Information Processing Systems	3	Variational Learning (VL) has recently gained popularity for training deep neural networks. Part of its empirical success can be explained by theories such as PAC-Bayes bounds, minimum description length and marginal likelihood, but little has been done to unravel the implicit regularization in play. Here, we analyze the implicit regularization of VL through the Edge of Stability (EoS) framework. EoS has previously been used to show that gradient descent can find flat solutions and we extend this result to show that VL can find even flatter solutions. This result is obtained by controlling the shape of the variational posterior as well as the number of posterior samples used during training. The derivation follows in a similar fashion as in the standard EoS literature for deep learning, by first deriving a result for a quadratic problem and then extending it to deep neural networks. We empirically validate these findings on a wide variety of large networks, such as ResNet and ViT, to find that the theoretical results closely match the empirical ones. Ours is the first work to analyze the EoS dynamics of VL.
128	b5c326ce9d365a4a4a137701aca50252809f58e7	2506.11305	10.48550/arXiv.2506.11305	2025	Don't Pay Attention	Mohammad Hammoud	2	arXiv.org	1	The Transformer has become the de facto standard for modern language models owing to its parallelizable training and effective autoregressive decoding. However, its fixed context window and the quadratic time and memory costs of its self-attention mechanism remain central bottlenecks. These constraints have revived interest in recurrent architectures that scale linearly with sequence length, but at the cost of reduced parallelism. In this paper, we introduce Avey, a new foundational architecture that breaks away from both attention and recurrence. Avey pairs a ranker with an autoregressive neural processor to select and contextualize only the most relevant tokens for any given token. Specifically, it decouples sequence length from context width, thus enabling effective and efficient processing of arbitrarily long sequences. Results show that Avey compares favorably to the Transformer across a variety of standard short-range NLP benchmarks, while significantly outperforming it on tasks requiring long-range dependency modeling.
129	92cd1a5246b6db4294ada7e2f6fca68d5abffd33	2506.10298		2025	GEARS H: Accurate machine-learned Hamiltonians for next-generation device-scale modeling	Anubhab Haldar	4		3	We introduce GEARS H, a state-of-the-art machine-learning Hamiltonian framework for large-scale electronic structure simulations. Using GEARS H, we present a statistical analysis of the hole concentration induced in defective $\mathrm{WSe}_2$ interfaced with Ni-doped amorphous $\mathrm{HfO}_2$ as a function of the Ni doping rate, system density, and Se vacancy rate in 72 systems ranging from 3326 to 4160 atoms-a quantity and scale of interface electronic structure calculation beyond the reach of conventional density functional theory codes and other machine-learning-based methods. We further demonstrate the versatility of our architecture by training models for a molecular system, 2D materials with and without defects, solid solution crystals, and bulk amorphous systems with covalent and ionic bonds. The mean absolute error of the inferred Hamiltonian matrix elements from the validation set is below 2.4 meV for all of these models. GEARS H outperforms other proposed machine-learning Hamiltonian frameworks, and our results indicate that machine-learning Hamiltonian methods, starting with GEARS H, are now production-ready techniques for DFT-accuracy device-scale simulation.
130	27d027b5f3e590e04b4c73334be2d6a53571d195	2506.04805	10.48550/arXiv.2506.04805	2025	Adaptive Preconditioners Trigger Loss Spikes in Adam	Zhiwei Bai	9	arXiv.org	14	Loss spikes commonly emerge during neural network training with the Adam optimizer across diverse architectures and scales, yet their underlying mechanism remains elusive. While previous explanations attribute these phenomena to sharper loss landscapes at lower loss, we show that landscape geometry alone is insufficient to explain the phenomenon. In this work, we pinpoint the root cause in the internal dynamics of Adam's second moment estimator. We identify a critical ``decoupling''mechanism where the adaptive preconditioner $v_t$ fails to track the instantaneous squared gradients $g_t^2$, causing the adaptive mechanism to effectively fail. This decoupling allows the preconditioner to decay autonomously despite rising gradients, which pushes the maximum eigenvalue of the preconditioned Hessian beyond the stability threshold $2/\eta$ for sustained periods, manifesting as dramatic loss spikes. Through a quadratic approximation analysis, we theoretically and experimentally characterize five distinct stages of spike evolution and propose a predictor for anticipating spikes based on gradient-directional curvature. We empirically find that the proposed loss spike mechanism, although derived from simplified models, generalizes well to practical scenarios ranging from small neural networks to large-scale Transformers.
131	a5b3fb345f5f87ba2b921d2f69e5e1611f51e43f	2506.01562	10.48550/arXiv.2506.01562	2025	Unpacking Softmax: How Temperature Drives Representation Collapse, Compression, and Generalization	Wojciech Masarczyk	6	arXiv.org	4	The softmax function is a fundamental building block of deep neural networks, commonly used to define output distributions in classification tasks or attention weights in transformer architectures. Despite its widespread use and proven effectiveness, its influence on learning dynamics and learned representations remains poorly understood, limiting our ability to optimize model behavior. In this paper, we study the pivotal role of the softmax function in shaping the model's representation. We introduce the concept of rank deficit bias - a phenomenon in which softmax-based deep networks find solutions of rank much lower than the number of classes. This bias depends on the softmax function's logits norm, which is implicitly influenced by hyperparameters or directly modified by softmax temperature. Furthermore, we demonstrate how to exploit the softmax dynamics to learn compressed representations or to enhance their performance on out-of-distribution data. We validate our findings across diverse architectures and real-world datasets, highlighting the broad applicability of temperature tuning in improving model performance. Our work provides new insights into the mechanisms of softmax, enabling better control over representation learning in deep neural networks.
132	0cabc7ad0df8b4c57043410b9fdeda7d16431b8d	2505.24333	10.48550/arXiv.2505.24333	2025	Two failure modes of deep transformers and how to avoid them: a unified theory of signal propagation at initialisation	Alessio Giorlandino	2	arXiv.org	17	Finding the right initialisation for neural networks is crucial to ensure smooth training and good performance. In transformers, the wrong initialisation can lead to one of two failure modes of self-attention layers: rank collapse, where all tokens collapse into similar representations, and entropy collapse, where highly concentrated attention scores lead to training instability. While previous work has studied different scaling regimes for transformers, an asymptotically exact, down-to-the constant prescription for how to initialise transformers has so far been lacking. Here, we provide an analytical theory of signal propagation through deep transformers with self-attention, layer normalisation, skip connections and MLP. Our theory yields a simple algorithm to compute trainability diagrams that identify the correct choice of initialisation hyper-parameters for a given architecture. We overcome the key challenge, an exact treatment of the self-attention layer, by establishing a formal parallel with the Random Energy Model from statistical physics. We also analyse gradients in the backward path and determine the regime where gradients vanish at initialisation. We demonstrate the versatility of our framework through three case studies. Our theoretical framework gives a unified perspective on the two failure modes of self-attention and gives quantitative predictions on the scale of both weights and residual connections that guarantee smooth training.
133	acd38ea36e308e025df918d0d5e096897cc08535	2505.21910	10.48550/arXiv.2505.21910	2025	Taming Transformer Without Using Learning Rate Warmup	Xianbiao Qi	8	International Conference on Learning Representations	9	Scaling Transformer to a large scale without using some technical tricks such as learning rate warump and using an obviously lower learning rate is an extremely challenging task, and is increasingly gaining more attention. In this paper, we provide a theoretical analysis for the process of training Transformer and reveal the rationale behind the model crash phenomenon in the training process, termed \textit{spectral energy concentration} of ${\bW_q}^{\top} \bW_k$, which is the reason for a malignant entropy collapse, where ${\bW_q}$ and $\bW_k$ are the projection matrices for the query and the key in Transformer, respectively. To remedy this problem, motivated by \textit{Weyl's Inequality}, we present a novel optimization strategy, \ie, making the weight updating in successive steps smooth -- if the ratio $\frac{\sigma_{1}(\nabla \bW_t)}{\sigma_{1}(\bW_{t-1})}$ is larger than a threshold, we will automatically bound the learning rate to a weighted multiple of $\frac{\sigma_{1}(\bW_{t-1})}{\sigma_{1}(\nabla \bW_t)}$, where $\nabla \bW_t$ is the updating quantity in step $t$. Such an optimization strategy can prevent spectral energy concentration to only a few directions, and thus can avoid malignant entropy collapse which will trigger the model crash. We conduct extensive experiments using ViT, Swin-Transformer and GPT, showing that our optimization strategy can effectively and stably train these Transformers without using learning rate warmup.
134	3f511e7d150ef7a69818510d998c49f4f4041f9c	2505.20524	10.48550/arXiv.2505.20524	2025	Towards Fully FP8 GEMM LLM Training at Scale	Alejandro Hernández-Cano	4	Neural Information Processing Systems	13	Despite the significant potential of FP8 data formats for large language model (LLM) pre-training, their adoption has been limited due to challenges in maintaining stability at scale. Existing approaches often rely on suboptimal fine-grained FP8 kernels or fall back to higher-precision matrix multiplications (GEMMs) in sensitive components, such as attention projections, compromising potential throughput gains. We introduce a new class of LLM architectures that, for the first time, support FP8 computation for all GEMMs within transformer blocks during both forward and backward passes. This enables unprecedented throughput gains, particularly at scale, while matching the downstream performance of standard BF16 training. Our architecture design reduces large outlier activations, promoting stable long-term FP8 training. In addition, we identify key metrics to monitor low-precision training and predict potential future divergences.
135	b24cf10e60ed3d57b57d7958bb3e9e5c7e348121	2505.16333	10.48550/arXiv.2505.16333	2025	Understanding Differential Transformer Unchains Pretrained Self-Attentions	Chaerin Kong	3	Neural Information Processing Systems	0	Differential Transformer has recently gained significant attention for its impressive empirical performance, often attributed to its ability to perform noise canceled attention. However, precisely how differential attention achieves its empirical benefits remains poorly understood. Moreover, Differential Transformer architecture demands large-scale training from scratch, hindering utilization of open pretrained weights. In this work, we conduct an in-depth investigation of Differential Transformer, uncovering three key factors behind its success: (1) enhanced expressivity via negative attention, (2) reduced redundancy among attention heads, and (3) improved learning dynamics. Based on these findings, we propose DEX, a novel method to efficiently integrate the advantages of differential attention into pretrained language models. By reusing the softmax attention scores and adding a lightweight differential operation on the output value matrix, DEX effectively incorporates the key advantages of differential attention while remaining lightweight in both training and inference. Evaluations confirm that DEX substantially improves the pretrained LLMs across diverse benchmarks, achieving significant performance gains with minimal adaptation data (<0.01%).
136	f6c0ed9a4e38593b3ab02b7d0aac8933db5b3c3a	2505.15325	10.1007/s11263-026-02841-z	2025	SoftHGNN: Soft Hypergraph Neural Networks for General Visual Recognition	Mengqi Lei	7	International Journal of Computer Vision	10	Visual recognition relies on understanding both the semantics of image tokens and the complex interactions among them. Mainstream self-attention methods, while effective at modeling global pair-wise relations, fail to capture high-order associations inherent in real-world scenes and often suffer from redundant computation. Hypergraphs extend conventional graphs by modeling high-order interactions and offer a promising framework for addressing these limitations. However, existing hypergraph neural networks typically rely on static and hard hyperedge assignments, which lead to redundant hyperedges and overlooking the continuity of visual semantics. In this work, we present Soft Hypergraph Neural Networks (SoftHGNN), a lightweight plug-and-play hypergraph computation method for late-stage semantic reasoning in existing vision pipelines. Our SoftHGNN introduces the concept of soft hyperedges, where each vertex is associated with hyperedges via continuous and differentiable participation weights rather than hard binary assignments. These weights are produced by measuring similarities between vertex features and a small set of learnable hyperedge prototypes, yielding input-adaptive and semantically rich soft hyperedges. Using soft hyperedges as the medium for message aggregation and dissemination, SoftHGNN enriches feature representations with high-order contextual associations. To further enhance efficiency when scaling up the number of soft hyperedges, we incorporate a sparse hyperedge selection mechanism that activates only the top-k important hyperedges, along with a load-balancing regularizer to ensure adequate and balanced hyperedge utilization. Experimental results across three tasks on five datasets demonstrate that SoftHGNN efficiently captures high-order associations in visual scenes, achieving significant performance improvements. The source code of our framework is available at: https://github.com/Mengqi-Lei/SoftHGNN.
137	a79374ac9141da7c584cda10ec6dc77d06bd22cf	2505.15080	10.48550/arXiv.2505.15080	2025	SUS backprop: linear backpropagation algorithm for long inputs in transformers	S. Pankov	2	arXiv.org	0	It is straightforward to design an unbiased gradient estimator that stochastically cuts the backpropagation flow through any part of a computational graph. By cutting the parts that have little effect on the computation, one can potentially save a significant amount of backpropagation computation in exchange for a minimal increase in the stochastic gradient variance, in some situations. Such a situation occurs in the attention mechanism of the transformer architecture. For long sequences, attention becomes the limiting factor, as its compute requirements increase quadratically with sequence length $n$. At the same time, most attention weights become very small, as most attention heads tend to connect a given token with only a small fraction of other tokens in the sequence. These weights become promising targets for cutting backpropagation. We propose a simple probabilistic rule controlled by a single parameter $c$ that cuts back-propagation through most attention weights, leaving at most $c$ interactions per token per attention head. This brings a factor of $c/n$ reduction in the compute required for the attention backpropagation, turning it from quadratic $O(n^2)$ to linear complexity $O(nc)$. We have empirically verified that, for a typical transformer model, cutting about $99\%$ of the attention gradient flow (i.e. choosing $c \sim 25-30$) results in relative gradient variance increase of only about $1\%$ for $n \sim 2000$, and it decreases with $n$. This approach is amenable to efficient sparse matrix implementation, thus being promising for making the cost of a backward pass negligible relative to the cost of a forward pass when training a transformer model on long sequences.
138	fd2fa79070ed3d0e5e29ed6527f625e61b75bbc6		10.1109/ICDE65448.2025.00318	2025	FlowFill: A Fast and Energy-Efficient Streaming Framework for Recovering Missing Values in Industrial Sensor Data	Hao Huang	2	IEEE International Conference on Data Engineering	0	Industrial systems such as wind turbines and nuclear facilities depend on continuous sensor monitoring to ensure operational safety and efficiency. However, missing data—caused by sensor malfunctions or environmental interference—can compromise real-time analyses, delaying critical maintenance decisions and system optimizations. Existing imputation methods often struggle to efficiently handle large data volumes in a streaming manner, typically requiring full access to historical time series and consuming excessive energy and processing time. To address these limitations, we propose FlowFill, a fast and energy-efficient streaming framework for recovering missing values in industrial sensor data. FlowFill leverages multiple dynamic sketches to capture local temporal patterns and encodes long-term and diverse data characteristics. Operating incrementally, FlowFill updates its sketches with new data samples, ensuring real-time adaptability. An attention mechanism dynamically prioritizes relevant sketches, dynamically weighting their contributions for precise imputations. This framework excels in computational efficiency, resilience to diverse missing patterns, and energy conservation, making it ideal for industrial applications that demand high-speed processing and energy efficiency. Experiments on industrial datasets demonstrate that FlowFill surpasses state-of-the-art methods in both accuracy and efficiency, establishing a new benchmark for streaming imputation in industrial contexts.
139	f42c257343f019456cbeb5e077d55ecf981957fb	2505.04738		2025	SetONet: A Set-Based Operator Network for Solving PDEs with Variable-Input Sampling	S. Tretiakov	3		2	Most neural-operator surrogates for PDEs inherit from DeepONet-style formulations the requirement that the input function be sampled at a fixed, ordered set of sensors. This assumption limits applicability to problems with variable sensor layouts, missing data, point sources, and sample-based representations of densities. We propose SetONet, which addresses this gap by recasting the operator input as an unordered set of coordinate-value observations and encoding it with permutation-invariant aggregation inside a standard branch-trunk operator network while preserving the DeepONet synthesis mechanism and lightweight end-to-end training. A structured variant, SetONet-Key, aggregates sensor information through learnable query tokens and a position-only key pathway, thereby decoupling sampling geometry from sensor values. The method is assessed on four classical operator-learning benchmarks under fixed layouts, variable layouts, and evaluation-time sensor drop-off, and on four problems with inherently unstructured point-cloud inputs, including heat conduction with multiple point sources, advection-diffusion, phase-screen diffraction, and optimal transport problems. In parameter-matched studies, SetONet-Key achieves lower error than the DeepONet baseline on fixed-sensor benchmarks and remains reliable when layouts vary or sensors are dropped at evaluation. Comparisons across pooling rules show that attention-based aggregation is typically more robust than mean or sum pooling. On the point-cloud problems, SetONet operates directly on the native input representation, without rasterization or multi-stage preprocessing, and outperforms the larger VIDON baseline.
140	0400f32963eac721227b74c6335bc1fd63446fba		10.1016/j.bspc.2024.107478	2025	Joint VMD and PSSVM optimization algorithm for seizure prediction based on ECG signals	Yun-Lin Chen	8	Biomedical Signal Processing and Control	3
141	fdf1ac9c0902dc9e2c127f4128c22b73277367c8	2504.16275	10.48550/arXiv.2504.16275	2025	Quantum Doubly Stochastic Transformers	Jannis Born	6	Neural Information Processing Systems	8	At the core of the Transformer, the softmax normalizes the attention matrix to be right stochastic. Previous research has shown that this often de-stabilizes training and that enforcing the attention matrix to be doubly stochastic (through Sinkhorn's algorithm) consistently improves performance across different tasks, domains and Transformer flavors. However, Sinkhorn's algorithm is iterative, approximative, non-parametric and thus inflexible w.r.t. the obtained doubly stochastic matrix (DSM). Recently, it has been proven that DSMs can be obtained with a parametric quantum circuit, yielding a novel quantum inductive bias for DSMs with no known classical analogue. Motivated by this, we demonstrate the feasibility of a hybrid classical-quantum doubly stochastic Transformer (QDSFormer) that replaces the softmax in the self-attention layer with a variational quantum circuit. We study the expressive power of the circuit and find that it yields more diverse DSMs that better preserve information than classical operators. Across multiple small-scale object recognition tasks, we find that our QDSFormer consistently surpasses both a standard ViT and other doubly stochastic Transformers. Beyond the Sinkformer, this comparison includes a novel quantum-inspired doubly stochastic Transformer (based on QR decomposition) that can be of independent interest. Our QDSFormer also shows improved training stability and lower performance variation suggesting that it may mitigate the notoriously unstable training of ViTs on small-scale data.
142	6020eff9434e0a17575a4ac82ac520b6d582a826		10.1609/aaai.v39i19.34211	2025	Enhancing Masked Time-Series Modeling via Dropping Patches	Tian-Yu Qiu	5	AAAI Conference on Artificial Intelligence	3	This paper explores how to enhance existing masked time-series modeling by randomly dropping sub-sequence level patches of time series. On this basis, a simple yet effective method named DropPatch is proposed, which has two remarkable advantages: 1) It improves the pre-training efficiency by a square-level advantage; 2) It provides additional advantages for modeling in scenarios such as in-domain, cross-domain, few-shot learning and cold start. This paper conducts comprehensive experiments to verify the effectiveness of the method and analyze its internal mechanism. Empirically, DropPatch strengthens the attention mechanism, reduces information redundancy and serves as an efficient means of data augmentation. Theoretically, it is proved that DropPatch slows down the rate at which the Transformer representations collapse into the rank-1 linear subspace by randomly dropping patches, thus optimizing the quality of the learned representations.
143	383952bfba2d5d164f35456fa025faf19f6b51b4	2504.04395	10.48550/arXiv.2504.04395	2025	Human-Level Competitive Pokémon via Scalable Offline Reinforcement Learning with Transformers	Jake Grigsby	5	arXiv.org	5	Competitive Pok\'emon Singles (CPS) is a popular strategy game where players learn to exploit their opponent based on imperfect information in battles that can last more than one hundred stochastic turns. AI research in CPS has been led by heuristic tree search and online self-play, but the game may also create a platform to study adaptive policies trained offline on large datasets. We develop a pipeline to reconstruct the first-person perspective of an agent from logs saved from the third-person perspective of a spectator, thereby unlocking a dataset of real human battles spanning more than a decade that grows larger every day. This dataset enables a black-box approach where we train large sequence models to adapt to their opponent based solely on their input trajectory while selecting moves without explicit search of any kind. We study a progression from imitation learning to offline RL and offline fine-tuning on self-play data in the hardcore competitive setting of Pok\'emon's four oldest (and most partially observed) game generations. The resulting agents outperform a recent LLM Agent approach and a strong heuristic search engine. While playing anonymously in online battles against humans, our best agents climb to rankings inside the top 10% of active players. All agent checkpoints, training details, datasets, and baselines are available at https://metamon.tech.
144	644aa8627063b33c3f48a6211f8286f05e4430ee	2504.03727	10.48550/arXiv.2504.03727	2025	Graph Transformer-Based Flood Susceptibility Mapping: Application to the French Riviera and Railway Infrastructure Under Climate Change	Sreenath Vemula	3	Journal of Hydrology: Regional Studies	7	Increasing flood frequency and severity due to climate change threatens infrastructure and demands improved susceptibility mapping techniques. While traditional machine learning (ML) approaches are widely used, they struggle to capture spatial dependencies and poor boundary delineation between susceptibility classes. This study introduces the first application of a graph transformer (GT) architecture for flood susceptibility mapping to the flood-prone French Riviera (e.g., 2020 Storm Alex) using topography, hydrology, geography, and environmental data. GT incorporates watershed topology using Laplacian positional encoders (PEs) and attention mechanisms. The developed GT model has an AUC-ROC (0.9739), slightly lower than XGBoost (0.9853). However, the GT model demonstrated better clustering and delineation with a higher Moran's I value (0.6119) compared to the random forest (0.5775) and XGBoost (0.5311) with p-value lower than 0.0001. Feature importance revealed a striking consistency across models, with elevation, slope, distance to channel, and convergence index being the critical factors. Dimensionality reduction on Laplacian PEs revealed partial clusters, indicating they could capture spatial information; however, their importance was lower than flood factors. Since climate and land use changes aggravate flood risk, susceptibility maps are developed for the 2050 year under different Representative Concentration Pathways (RCPs) and railway track vulnerability is assessed. All RCP scenarios revealed increased area across susceptibility classes, except for the very low category. RCP 8.5 projections indicate that 17.46% of the watershed area and 54% of railway length fall within very-high susceptible zones, compared to 6.19% and 35.61%, respectively, under current conditions. The developed maps can be integrated into a multi-hazard framework.
145	18cb0d9487f09c8121127e2c6d08609fed831a16	2504.00149	10.48550/arXiv.2504.00149	2025	Towards Precise Action Spotting: Addressing Temporal Misalignment in Labels with Dynamic Label Assignment	Masato Tamura	1	arXiv.org	0	Precise action spotting has attracted considerable attention due to its promising applications. While existing methods achieve substantial performance by employing well-designed model architecture, they overlook a significant challenge: the temporal misalignment inherent in ground-truth labels. This misalignment arises when frames labeled as containing events do not align accurately with the actual event times, often as a result of human annotation errors or the inherent difficulties in precisely identifying event boundaries across neighboring frames. To tackle this issue, we propose a novel dynamic label assignment strategy that allows predictions to have temporal offsets from ground-truth action times during training, ensuring consistent event spotting. Our method extends the concept of minimum-cost matching, which is utilized in the spatial domain for object detection, to the temporal domain. By calculating matching costs based on predicted action class scores and temporal offsets, our method dynamically assigns labels to the most likely predictions, even when the predicted times of these predictions deviate from ground-truth times, alleviating the negative effects of temporal misalignment in labels. We conduct extensive experiments and demonstrate that our method achieves state-of-the-art performance, particularly in conditions where events are visually distinct and temporal misalignment in labels is common.
146	74551012014d4206bdd31f0a11c78f5b80ed9753	2503.18337	10.1109/CVPR52734.2025.01876	2025	Coeff-Tuning: A Graph Filter Subspace View for Tuning Attention-Based Large Models	Zi-Chen Miao	3	Computer Vision and Pattern Recognition	10	Transformer-based large pre-trained models have shown remarkable generalization ability, and various parameter-efficient fine-tuning (PEFT) methods have been proposed to customize these models on downstream tasks with minimal computational and memory budgets. Previous PEFT methods are primarily designed from a tensor-decomposition perspective that tries to effectively tune the linear transformation by finding the smallest subset of parameters to train. Our study adopts an orthogonal view by representing the attention operation as a graph convolution and formulating the multi-head attention maps as a convolutional filter subspace, with each attention map as a subspace element. In this paper, we propose to tune the large pre-trained transformers by learning a small set of combination coefficients that construct a more expressive filter subspace from the original multi-head attention maps. We show analytically and experimentally that the tuned filter subspace can effectively expand the feature space of the multi-head attention and further enhance the capacity of transformers. We further stabilize the fine-tuning with a residual parameterization of the tunable subspace coefficients, and enhance the generalization with a regularization design by directly applying dropout on the tunable coefficient during training. The tunable coefficients take a tiny number of parameters and can be combined with previous PEFT methods in a plug-and-play manner. Extensive experiments show that our approach achieves superior performances than PEFT baselines with neglectable additional parameters.1
147	4356b46e5dd1a4ebd579b1cd6eb3eeedddd5a65c	2503.11651	10.1109/CVPR52734.2025.00499	2025	VGGT: Visual Geometry Grounded Transformer	Jian-Yuan Wang	6	Computer Vision and Pattern Recognition	2112	We present VGGT, a feed-forward neural network that directly infers all key 3D attributes of a scene, including camera parameters, point maps, depth maps, and 3D point tracks, from one, a few, or hundreds of its views. This approach is a step forward in 3D computer vision, where models have typically been constrained to and specialized for single tasks. It is also simple and efficient, reconstructing images in under one second, and still outperforming alternatives that require post-processing with visual geometry optimization techniques. The network achieves state-of-the-art results in multiple 3D tasks, including camera parameter estimation, multi-view depth estimation, dense point cloud reconstruction, and 3D point tracking. We also show that using pretrained VGGT as a feature backbone significantly enhances downstream tasks, such as non-rigid point tracking and feed-forward novel view synthesis. Code and models are publicly available at https://github.com/facebookresearch/vggt.
148	1e178f0c5bb9709ae5c7bdb60ecd76f00b0fcd86	2503.10622	10.1109/CVPR52734.2025.01388	2025	Transformers without Normalization	Jia-Cheng Zhu	5	Computer Vision and Pattern Recognition	229	Normalization layers are ubiquitous in modern neural networks and have long been considered essential. This work demonstrates that Transformers without normalization can achieve the same or better performance using a remarkably simple technique. We introduce Dynamic Tanh (DyT), an element-wise operation DyT(x) = tanh(αx), as a dropin replacement for normalization layers in Transformers. DyT is inspired by the observation that layer normalization in Transformers often produces tanh-like, S-shaped input-output mappings. By incorporating DyT, Transformers without normalization can match or exceed the performance of their normalized counterparts, mostly without hyperparameter tuning. We validate the effectiveness of Transformers with DyT across diverse settings, from recognition to generation, supervised to self-supervised learning, and computer vision to language models. These findings challenge the conventional understanding that normalization layers are indispensable in modern neural networks, and offer new insights into their role in deep networks.
149	843a3924de7a24689bc080362e9ddd610fd25507	2503.09543	10.48550/arXiv.2503.09543	2025	PolyPythias: Stability and Outliers across Fifty Language Model Pre-Training Runs	Oskar van der Wal	7	International Conference on Learning Representations	40	The stability of language model pre-training and its effects on downstream performance are still understudied. Prior work shows that the training process can yield significantly different results in response to slight variations in initial conditions, e.g., the random seed. Crucially, the research community still lacks sufficient resources and tools to systematically investigate pre-training stability, particularly for decoder-only language models. We introduce the PolyPythias, a set of 45 new training runs for the Pythia model suite: 9 new seeds across 5 model sizes, from 14M to 410M parameters, resulting in about 7k new checkpoints that we release. Using these new 45 training runs, in addition to the 5 already available, we study the effects of different initial conditions determined by the seed -- i.e., parameters' initialisation and data order -- on (i) downstream performance, (ii) learned linguistic representations, and (iii) emergence of training phases. In addition to common scaling behaviours, our analyses generally reveal highly consistent training dynamics across both model sizes and initial conditions. Further, the new seeds for each model allow us to identify outlier training runs and delineate their characteristics. Our findings show the potential of using these methods to predict training stability.
150	0ece8e945fb157c9d8340f2c6f2959cf9226f860	2502.20750	10.48550/arXiv.2502.20750	2025	Mitigating Hallucinations in Large Vision-Language Models by Adaptively Constraining Information Flow	Jia-Qi Bai	7	AAAI Conference on Artificial Intelligence	17	Large vision-language models show tremendous potential in understanding visual information through human languages. However, they are prone to suffer from object hallucination, i.e., the generated image descriptions contain objects that do not exist in the image. In this paper, we reveal that object hallucination can be attributed to overconfidence in irrelevant visual features when soft visual tokens map to the LLM's word embedding space. Specifically, by figuring out the semantic similarity between visual tokens and LLM's word embedding, we observe that the smoothness of similarity distribution strongly correlates with the emergence of object hallucinations. To mitigate hallucinations, we propose using the Variational Information Bottleneck (VIB) to alleviate overconfidence by introducing stochastic noise, facilitating the constraining of irrelevant information. Furthermore, we propose an entropy-based noise-controlling strategy to enable the injected noise to be adaptively constrained regarding the smoothness of the similarity distribution. We adapt the proposed AdaVIB across distinct model architectures. Experimental results demonstrate that the proposed AdaVIB mitigates object hallucinations by effectively alleviating the overconfidence in irrelevant visual features, with consistent improvements on two object hallucination benchmarks.
151	e1d3877653128923851f21f74ea0ec04ea968e90	2502.17599	10.48550/arXiv.2502.17599	2025	MEDA: Dynamic KV Cache Allocation for Efficient Multimodal Long-Context Inference	Zhong-Wei Wan	6	North American Chapter of the Association for Computational Linguistics	42	Long-context Multimodal Large Language Models (MLLMs) that incorporate long text-image and text-video modalities, demand substantial resources as their multimodal Key-Value (KV) caches grow with increasing input lengths, challenging inference efficiency. Existing methods for KV cache compression, in both text-only and multimodal LLMs, have neglected attention density variations across layers, thus often adopting uniform or progressive reduction strategies for layer-wise cache allocation. In this work, we propose MEDA, a dynamic layer-wise KV cache allocation method for efficient multimodal long-context inference. As its core, MEDA utilizes cross-modal attention entropy to determine the KV cache size at each MLLMs layer. Given the dynamically allocated KV cache size at each layer, MEDA also employs a KV pair selection scheme to identify which KV pairs to select and a KV pair merging strategy that merges the selected and non-selected ones to preserve information from the entire context. MEDA achieves up to 72% KV cache memory reduction and 2.82 times faster decoding speed, while maintaining or enhancing performance on various multimodal tasks in long-context settings, including multi-images and long-video scenarios. Our code is released at https://github.com/AIoT-MLSys-Lab/MEDA.
152	2bd2d0df591cfb1906651e32099b2b60f936ab27	2502.16570		2025	Entropy-Lens: Uncovering Decision Strategies in LLMs	Riccardo Ali	4		8	In large language models (LLMs), each block operates on the residual stream to map input token sequences to output token distributions. However, most of the interpretability literature focuses on internal latent representations, leaving token-space dynamics underexplored. The high dimensionality and categoricity of token distributions hinder their analysis, as standard statistical descriptors are not suitable. We show that the entropy of logit-lens predictions overcomes these issues. In doing so, it provides a per-layer scalar, permutation-invariant metric. We introduce Entropy-Lens to distill the token-space dynamics of the residual stream into a low-dimensional signal. We call this signal the entropy profile. We apply our method to a variety of model sizes and families, showing that (i) entropy profiles uncover token prediction dynamics driven by expansion and pruning strategies; (ii) these dynamics are family-specific and invariant under depth rescaling; (iii) they are characteristic of task type and output format; (iv) these strategies have unequal impact on downstream performance, with the expansion strategy usually being more critical. Ultimately, our findings further enhance our understanding of the residual stream, enabling a granular assessment of how information is processed across model depth.
153	759501d8f58c74d2767f882086d63de588c791b4	2502.14704	10.48550/arXiv.2502.14704	2025	Not All Data are Good Labels: On the Self-supervised Labeling for Time Series Forecasting	Yuxuan Yang	6	Neural Information Processing Systems	5	Time Series Forecasting (TSF) is a crucial task in various domains, yet existing TSF models rely heavily on high-quality data and insufficiently exploit all available data. This paper explores a novel self-supervised approach to re-label time series datasets by inherently constructing candidate datasets. During the optimization of a simple reconstruction network, intermediates are used as pseudo labels in a self-supervised paradigm, improving generalization for any predictor. We introduce the Self-Correction with Adaptive Mask (SCAM), which discards overfitted components and selectively replaces them with pseudo labels generated from reconstructions. Additionally, we incorporate Spectral Norm Regularization (SNR) to further suppress overfitting from a loss landscape perspective. Our experiments on eleven real-world datasets demonstrate that SCAM consistently improves the performance of various backbone models. This work offers a new perspective on constructing datasets and enhancing the generalization of TSF models through self-supervised learning. The code is available at https://github.com/SuDIS-ZJU/SCAM.
154	a94021bc75d7d0694be5c1a43cb17b5a3af0ae39	2502.11066	10.48550/arXiv.2502.11066	2025	CARMA: Enhanced Compositionality in LLMs via Advanced Regularisation and Mutual Information Alignment	N. Aljaafari	3	Conference on Empirical Methods in Natural Language Processing	4	Large language models (LLMs) struggle with compositional generalisation, limiting their ability to systematically combine learned components to interpret novel inputs. While architectural modifications, fine-tuning, and data augmentation improve compositionality, they often have limited adaptability, face scalability constraints, or yield diminishing returns on real data. To address this, we propose CARMA, an intervention that enhances the stability and robustness of compositional reasoning in LLMs while preserving fine-tuned performance. CARMA employs mutual information regularisation and layer-wise stability constraints to mitigate feature fragmentation, ensuring structured representations persist across and within layers. We evaluate CARMA on inverse dictionary modelling and sentiment classification, measuring its impact on semantic consistency, performance stability, and robustness to lexical perturbations. Results show that CARMA reduces the variability introduced by fine-tuning, stabilises token representations, and improves compositional reasoning. While its effectiveness varies across architectures, CARMA's key strength lies in reinforcing learned structures rather than introducing new capabilities, making it a scalable auxiliary method. These findings suggest that integrating CARMA with fine-tuning can improve compositional generalisation while maintaining task-specific performance in LLMs.
155	b1e62f72336064184edb998eb39115fc9b6a6243	2502.02732		2025	Peri-LN: Revisiting Normalization Layer in the Transformer Architecture	Jeonghoon Kim	10	International Conference on Machine Learning	41	Selecting a layer normalization (LN) strategy that stabilizes training and speeds convergence in Transformers remains difficult, even for today's large language models (LLM). We present a comprehensive analytical foundation for understanding how different LN strategies influence training dynamics in large-scale Transformers. Until recently, Pre-LN and Post-LN have long dominated practices despite their limitations in large-scale training. However, several open-source models have recently begun silently adopting a third strategy without much explanation. This strategy places normalization layer peripherally around sublayers, a design we term Peri-LN. While Peri-LN has demonstrated promising performance, its precise mechanisms and benefits remain almost unexplored. Our in-depth analysis delineates the distinct behaviors of LN strategies, showing how each placement shapes activation variance and gradient propagation. To validate our theoretical insight, we conduct extensive experiments on Transformers up to $3.2$B parameters, showing that Peri-LN consistently achieves more balanced variance growth, steadier gradient flow, and convergence stability. Our results suggest that Peri-LN warrants broader consideration for large-scale Transformer architectures, providing renewed insights into the optimal placement of LN.
156	400f68671e9b958350d0400229a8c3d75d983434	2501.14577	10.48550/arXiv.2501.14577	2025	ZETA: Leveraging Z-order Curves for Efficient Top-k Attention	Qiuhao Zeng	7	International Conference on Learning Representations	12	Over recent years, the Transformer has become a fundamental building block for sequence modeling architectures. Yet at its core is the use of self-attention, whose memory and computational cost grow quadratically with the sequence length $N$, rendering it prohibitively expensive for long sequences. A promising approach is top-$k$ attention, which selects only the $k$ most relevant tokens and achieves performance comparable to vanilla self-attention while significantly reducing space and computational demands. However, causal masks require the current query token to only attend to past tokens, preventing the existing top-$k$ attention method from efficiently searching for the most relevant tokens in parallel, thereby limiting training efficiency. In this work, we propose ZETA, leveraging \textbf{Z}-Order Curves for \textbf{E}fficient \textbf{T}op-$k$ \textbf{A}ttention, to enable parallel querying of past tokens for entire sequences. % in both space and time complexity of $\mathcal{O}(N \log N)$. We first theoretically show that the choice of key and query dimensions involves a trade-off between the curse of dimensionality and the preservation of relative distances after projection. In light of this insight, we propose reducing the dimensionality of keys and queries in contrast to values and further leverage $Z$-order curves to map low-dimensional keys and queries into \emph{one}-dimensional space, which permits parallel sorting, thereby largely improving the efficiency for top-$k$ token selection. Experimental results demonstrate that ZETA matches the performance of standard attention on the synthetic \textsc{Multi-Query Associative Recall} task and outperforms attention and its variants on \textsc{Long Range Arena} and \textsc{WikiText-103} language modeling.
157	359ec89fe7e4d72b0fe5801a3507d74942f3c08d	2501.06842	10.48550/arXiv.2501.06842	2025	SPAM: Spike-Aware Adam with Momentum Reset for Stable LLM Training	Tian-Jin Huang	6	International Conference on Learning Representations	38	Large Language Models (LLMs) have demonstrated exceptional performance across diverse tasks, yet their training remains highly resource-intensive and susceptible to critical challenges such as training instability. A predominant source of this instability stems from gradient and loss spikes, which disrupt the learning process, often leading to costly interventions like checkpoint recovery and experiment restarts, further amplifying inefficiencies. This paper presents a comprehensive investigation into gradient spikes observed during LLM training, revealing their prevalence across multiple architectures and datasets. Our analysis shows that these spikes can be up to $1000\times$ larger than typical gradients, substantially deteriorating model performance. To address this issue, we propose Spike-Aware Adam with Momentum Reset SPAM, a novel optimizer designed to counteract gradient spikes through momentum reset and spike-aware gradient clipping. Extensive experiments, including both pre-training and fine-tuning, demonstrate that SPAM consistently surpasses Adam and its variants across various tasks, including (1) LLM pre-training from 60M to 1B, (2) 4-bit LLM pre-training,(3) reinforcement learning, and (4) Time Series Forecasting. Additionally, SPAM facilitates memory-efficient training by enabling sparse momentum, where only a subset of momentum terms are maintained and updated. When operating under memory constraints, SPAM outperforms state-of-the-art memory-efficient optimizers such as GaLore and Adam-Mini. Our work underscores the importance of mitigating gradient spikes in LLM training and introduces an effective optimization strategy that enhances both training stability and resource efficiency at scale. Code is available at https://github.com/TianjinYellow/SPAM-Optimizer.git
158	db13b49ede9b44ccd95ace924a1a7ef81bbd4a75	2412.21124	10.48550/arXiv.2412.21124	2024	Adaptive Batch Size Schedules for Distributed Training of Language Models with Data and Model Parallelism	Tim Tsz-Kit Lau	5	CPAL	6	An appropriate choice of batch sizes in large-scale model training is crucial, yet it involves an intrinsic yet inevitable dilemma: large-batch training improves training efficiency in terms of memory utilization, while generalization performance often deteriorates due to small amounts of gradient noise. Despite this dilemma, the common practice of choosing batch sizes in language model training often prioritizes training efficiency -- employing either constant large sizes with data parallelism or implementing batch size warmup schedules. However, such batch size schedule designs remain heuristic and often fail to adapt to training dynamics, presenting the challenge of designing adaptive batch size schedules. Given the abundance of available datasets and the data-hungry nature of language models, data parallelism has become an indispensable distributed training paradigm, enabling the use of larger batch sizes for gradient computation. However, vanilla data parallelism requires replicas of model parameters, gradients, and optimizer states at each worker, which prohibits training larger models with billions of parameters. To optimize memory usage, more advanced parallelism strategies must be employed. In this work, we propose general-purpose and theoretically principled adaptive batch size schedules compatible with data parallelism and model parallelism. We develop a practical implementation with PyTorch Fully Sharded Data Parallel, facilitating the pretraining of language models of different sizes. We empirically demonstrate that our proposed approaches outperform constant batch sizes and heuristic batch size warmup schedules in the pretraining of models in the Llama 2 family, with particular focus on smaller models with up to 3 billion parameters. We also establish theoretical convergence guarantees for such adaptive batch size schedules with Adam for general smooth nonconvex objectives.
159	a05a03089e1f8b7aa7dd0e2d140ca03dd04dcc56	2411.11188	10.48550/arXiv.2411.11188	2024	AMAGO-2: Breaking the Multi-Task Barrier in Meta-Reinforcement Learning with Transformers	Jake Grigsby	6	Neural Information Processing Systems	18	Language models trained on diverse datasets unlock generalization by in-context learning. Reinforcement Learning (RL) policies can achieve a similar effect by meta-learning within the memory of a sequence model. However, meta-RL research primarily focuses on adapting to minor variations of a single task. It is difficult to scale towards more general behavior without confronting challenges in multi-task optimization, and few solutions are compatible with meta-RL's goal of learning from large training sets of unlabeled tasks. To address this challenge, we revisit the idea that multi-task RL is bottlenecked by imbalanced training losses created by uneven return scales across different tasks. We build upon recent advancements in Transformer-based (in-context) meta-RL and evaluate a simple yet scalable solution where both an agent's actor and critic objectives are converted to classification terms that decouple optimization from the current scale of returns. Large-scale comparisons in Meta-World ML45, Multi-Game Procgen, Multi-Task POPGym, Multi-Game Atari, and BabyAI find that this design unlocks significant progress in online multi-task adaptation and memory problems without explicit task labels.
160	8f6339811a4765c41e507a7d6676c69835422144	2411.01443	10.48550/arXiv.2411.01443	2024	Activating Self-Attention for Multi-Scene Absolute Pose Regression	Miso Lee	3	Neural Information Processing Systems	4	Multi-scene absolute pose regression addresses the demand for fast and memory-efficient camera pose estimation across various real-world environments. Nowadays, transformer-based model has been devised to regress the camera pose directly in multi-scenes. Despite its potential, transformer encoders are underutilized due to the collapsed self-attention map, having low representation capacity. This work highlights the problem and investigates it from a new perspective: distortion of query-key embedding space. Based on the statistical analysis, we reveal that queries and keys are mapped in completely different spaces while only a few keys are blended into the query region. This leads to the collapse of the self-attention map as all queries are considered similar to those few keys. Therefore, we propose simple but effective solutions to activate self-attention. Concretely, we present an auxiliary loss that aligns queries and keys, preventing the distortion of query-key space and encouraging the model to find global relations by self-attention. In addition, the fixed sinusoidal positional encoding is adopted instead of undertrained learnable one to reflect appropriate positional clues into the inputs of self-attention. As a result, our approach resolves the aforementioned problem effectively, thus outperforming existing methods in both outdoor and indoor scenes.
161	d589cbb383f6fb0d0ef2f2177fc11a41491511db	2411.00999	10.48550/arXiv.2411.00999	2024	Normalization Layer Per-Example Gradients are Sufficient to Predict Gradient Noise Scale in Transformers	Gavia Gray	4	Neural Information Processing Systems	7	Per-example gradient norms are a vital ingredient for estimating gradient noise scale (GNS) with minimal variance. Observing the tensor contractions required to compute them, we propose a method with minimal FLOPs in 3D or greater tensor regimes by simultaneously computing the norms while computing the parameter gradients. Using this method we are able to observe the GNS of different layers at higher accuracy than previously possible. We find that the total GNS of contemporary transformer models is predicted well by the GNS of only the normalization layers. As a result, focusing only on the normalization layer, we develop a custom kernel to compute the per-example gradient norms while performing the LayerNorm backward pass with zero throughput overhead. Tracking GNS on only those layers, we are able to guide a practical batch size schedule that reduces training time by 18% on a Chinchilla-optimal language model.
162	09e5531baec6d726f426428ba58b16c9c6e3e228	2410.21676	10.48550/arXiv.2410.21676	2024	How Does Critical Batch Size Scale in Pre-training?	Hanlin Zhang	8	International Conference on Learning Representations	79	Training large-scale models under given resources requires careful design of parallelism strategies. In particular, the efficiency notion of critical batch size (CBS), concerning the compromise between time and compute, marks the threshold beyond which greater data parallelism leads to diminishing returns. To operationalize it, we propose a measure of CBS and pre-train a series of auto-regressive language models, ranging from 85 million to 1.2 billion parameters, on the C4 dataset. Through extensive hyper-parameter sweeps and careful control of factors such as batch size, momentum, and learning rate along with its scheduling, we systematically investigate the impact of scale on CBS. Then we fit scaling laws with respect to model and data sizes to decouple their effects. Overall, our results demonstrate that CBS scales primarily with data size rather than model size, a finding we justify theoretically through the analysis of infinite-width limits of neural networks and infinite-dimensional least squares regression. Of independent interest, we highlight the importance of common hyper-parameter choices and strategies for studying large-scale pre-training beyond fixed training durations.
163	bb9864391fb1c69f8b3240205a8a8ac65b2b6d07	2410.16682	10.48550/arXiv.2410.16682	2024	Methods of improving LLM training stability	Oleg Rybakov	5	arXiv.org	16	Training stability of large language models(LLMs) is an important research topic. Reproducing training instabilities can be costly, so we use a small language model with 830M parameters and experiment with higher learning rates to force models to diverge. One of the sources of training instability is the growth of logits in attention layers. We extend the focus of the previous work and look not only at the magnitude of the logits but at all outputs of linear layers in the Transformer block. We observe that with a high learning rate the L2 norm of all linear layer outputs can grow with each training step and the model diverges. Specifically we observe that QKV, Proj and FC2 layers have the largest growth of the output magnitude. This prompts us to explore several options: 1) apply layer normalization not only after QK layers but also after Proj and FC2 layers too; 2) apply layer normalization after the QKV layer (and remove pre normalization). 3) apply QK layer normalization together with softmax capping. We show that with the last two methods we can increase learning rate by 1.5x (without model divergence) in comparison to an approach based on QK layer normalization only. Also we observe significant perplexity improvements for all three methods in comparison to the baseline model.
164	d6fbe7f4523bee8fa814694897347ecdc7d06944	2410.13835	10.48550/arXiv.2410.13835	2024	Active-Dormant Attention Heads: Mechanistically Demystifying Extreme-Token Phenomena in LLMs	Tian-Yu Guo	6	arXiv.org	62	Practitioners have consistently observed three puzzling phenomena in transformer-based large language models (LLMs): attention sinks, value-state drains, and residual-state peaks, collectively referred to as extreme-token phenomena. These phenomena are characterized by certain so-called"sink tokens"receiving disproportionately high attention weights, exhibiting significantly smaller value states, and having much larger residual-state norms than those of other tokens. These extreme tokens give rise to various challenges in LLM inference, quantization, and interpretability. We elucidate the mechanisms behind extreme-token phenomena. First, we show that these phenomena arise in very simple architectures -- transformers with one to three layers -- trained on a toy model, the Bigram-Backcopy (BB) task. In this setting, we identify an active-dormant mechanism, where attention heads become sinks for specific input domains while remaining non-sinks for others. Our theoretical analysis of the training dynamics reveals that these phenomena are driven by a mutual reinforcement mechanism. Building on these insights, we propose strategies to mitigate extreme-token phenomena during pretraining, including replacing softmax with ReLU and Adam with SGD. Next, we extend our analysis to pretrained LLMs, including Llama and OLMo, showing that many attention heads exhibit a similar active-dormant mechanism as in the BB task, and that the mutual reinforcement mechanism also governs the emergence of extreme-token phenomena during LLM pretraining. Our results reveal that many of the static and dynamic properties of extreme-token phenomena predicted by the BB task align with observations in pretrained LLMs.
165	80d86aea613a91272a27c81c2d6a43763da0bece	2410.13060		2024	AERO: Entropy-Guided Framework for Private LLM Inference	Nandan Kumar Jha	2		6	Privacy-preserving computation enables language model inference directly on encrypted data yet suffers from prohibitive latency and communication overheads, primarily due to nonlinear functions. Removing nonlinearities, however, can trigger one of two failure modes restricting the potential for nonlinearity removal: entropy collapse in deeper layers, which destabilizes training, and entropic overload in early layers, causing under-utilization of attention heads. To address these challenges, we introduce AERO, an entropy-guided framework to strategically eliminates costly nonlinear operations from transformer architectures, which employs an adaptive recalibration through a head-wise entropy regularizer with learnable per-head strengths, enabling each head to adjust its entropy level while penalizing extreme entropies and fostering functional diversity through a tolerance margin. Experiments show AERO can save 3.4$\times$ communication and 1.4$\times$ latency, without any performance penalty.
166	26bd84d181af3812a62bc03ba005e223471a3bad	2410.11087	10.48550/arXiv.2410.11087	2024	Locality Alignment Improves Vision-Language Models	Ian Covert	4	International Conference on Learning Representations	17	Vision language models (VLMs) have seen growing adoption in recent years, but many still struggle with basic spatial reasoning errors. We hypothesize that this is due to VLMs adopting pre-trained vision backbones, specifically vision transformers (ViTs) trained with image-level supervision and minimal inductive biases. Such models may fail to encode the class contents at each position in the image, and our goal is to resolve this with a vision backbone that effectively captures both local and global image semantics. Our main insight is that we do not require new supervision to learn this capability - pre-trained models contain significant knowledge of local semantics that we can extract and use for scalable self-supervision. We propose a new efficient post-training stage for ViTs called locality alignment and a novel fine-tuning procedure called MaskEmbed that uses a masked reconstruction loss to learn semantic contributions for each image patch. We first evaluate locality alignment with a vision-only benchmark, finding that it improves a model's performance at patch-level semantic segmentation, especially for strong backbones trained with image-caption pairs (e.g., CLIP and SigLIP). We then train a series of VLMs with and without locality alignment, and show that locality-aligned backbones improve performance across a range of benchmarks, particularly ones that involve spatial understanding (e.g., RefCOCO, OCID-Ref, TallyQA, VSR, AI2D). Overall, we demonstrate that we can efficiently learn local semantic extraction via a locality alignment stage, and that this procedure benefits VLM training recipes that use off-the-shelf vision backbones.
167	c4b1a89de59602a3a2728f2705be6dcd4bfaa77f	2410.10781	10.48550/arXiv.2410.10781	2024	When Attention Sink Emerges in Language Models: An Empirical View	Xiangming Gu	8	International Conference on Learning Representations	237	Language Models (LMs) assign significant attention to the first token, even if it is not semantically important, which is known as attention sink. This phenomenon has been widely adopted in applications such as streaming/long context generation, KV cache optimization, inference acceleration, model quantization, and others. Despite its widespread use, a deep understanding of attention sink in LMs is still lacking. In this work, we first demonstrate that attention sinks exist universally in LMs with various inputs, even in small models. Furthermore, attention sink is observed to emerge during the LM pre-training, motivating us to investigate how optimization, data distribution, loss function, and model architecture in LM pre-training influence its emergence. We highlight that attention sink emerges after effective optimization on sufficient training data. The sink position is highly correlated with the loss function and data distribution. Most importantly, we find that attention sink acts more like key biases, storing extra attention scores, which could be non-informative and not contribute to the value computation. We also observe that this phenomenon (at least partially) stems from tokens' inner dependence on attention scores as a result of softmax normalization. After relaxing such dependence by replacing softmax attention with other attention operations, such as sigmoid attention without normalization, attention sinks do not emerge in LMs up to 1B parameters. The code is available at https://github.com/sail-sg/Attention-Sink.
168	a7563b8f0dc36ecdaea7c22441e5242a1029cf4a	2410.09637	10.48550/arXiv.2410.09637	2024	ReLU's Revival: On the Entropic Overload in Normalization-Free Large Language Models	Nandan Kumar Jha	2	arXiv.org	4	LayerNorm is a critical component in modern large language models (LLMs) for stabilizing training and ensuring smooth optimization. However, it introduces significant challenges in mechanistic interpretability, outlier feature suppression, faithful signal propagation, and computational and communication complexity of private inference. This work explores desirable activation functions in normalization-free decoder-only LLMs. Contrary to the conventional preference for the GELU in transformer-based models, our empirical findings demonstrate an {\em opposite trend} -- ReLU significantly outperforms GELU in LayerNorm-free models, leading to an {\bf 8.2\%} perplexity improvement. We discover a key issue with GELU, where early layers experience entropic overload, leading to the under-utilization of the representational capacity of attention heads. This highlights that smoother activations like GELU are {\em ill-suited} for LayerNorm-free architectures, whereas ReLU's geometrical properties -- specialization in input space and intra-class selectivity -- lead to improved learning dynamics and better information retention in the absence of LayerNorm. This study offers key insights for optimizing transformer architectures where LayerNorm introduces significant challenges. The code and implementation are available at https://github.com/Nandan91/relu-revival-normfree
169	8139791602a16401f90d8e9eed5a09e0e7d02a9d		10.51682/jiscom.v5i2.47	2024	Grammatical Error Checker for Text and Speech using NLP and Speech Recognition in LLMs	Abdul Haseeb	4	JOURNAL OF INTELLIGENT SYSTEMS AND COMPUTING	0	With the rapid growth of Artificial Intelligence (AI) and Machine Learning (ML), the ways in which we perform many of our day-to-day tasks have changed. The process of documentation has become simpler due to new-age tools that leverage technologies such as AI and ML to automate tasks such as searching for grammatical errors in sentences and spelling errors in words within a text document. Speech recognition is being used in the pursuit of automating tasks and reducing dependency on humans. Natural Language Processing, or NLP is used by Large Language Models (LLMs) to process inputs in the form of text or speech for a multitude of tasks such as text classification and language translation. This paper begins with an explanation of NLP and later provides an overview of speech recognition.
170	89a6e57f59fdfae4e8ae16e0753cc29b6b996001	2410.05052	10.48550/arXiv.2410.05052	2024	Initialization of Large Language Models via Reparameterization to Mitigate Loss Spikes	Kosuke Nishida	3	Conference on Empirical Methods in Natural Language Processing	7	Loss spikes, a phenomenon in which the loss value diverges suddenly, is a fundamental issue in the pre-training of large language models. This paper supposes that the non-uniformity of the norm of the parameters is one of the causes of loss spikes. Here, in training of neural networks, the scale of the gradients is required to be kept constant throughout the layers to avoid the vanishing and exploding gradients problem. However, to meet these requirements in the Transformer model, the norm of the model parameters must be non-uniform, and thus, parameters whose norm is smaller are more sensitive to the parameter update. To address this issue, we propose a novel technique, weight scaling as reparameterization (WeSaR). WeSaR introduces a gate parameter per parameter matrix and adjusts it to the value satisfying the requirements. Because of the gate parameter, WeSaR sets the norm of the original parameters uniformly, which results in stable training. Experimental results with the Transformer decoders consisting of 130 million, 1.3 billion, and 13 billion parameters showed that WeSaR stabilizes and accelerates training and that it outperformed compared methods including popular initialization methods.
171	8324e86cc7fae8fbb7f980bdef0665531f0d6f46	2410.04801	10.48550/arXiv.2410.04801	2024	Improving Image Clustering with Artifacts Attenuation via Inference-Time Attention Engineering	Kazumoto Nakamura	5	Asian Conference on Computer Vision	3	The goal of this paper is to improve the performance of pretrained Vision Transformer (ViT) models, particularly DINOv2, in image clustering task without requiring re-training or fine-tuning. As model size increases, high-norm artifacts anomaly appears in the patches of multi-head attention. We observe that this anomaly leads to reduced accuracy in zero-shot image clustering. These artifacts are characterized by disproportionately large values in the attention map compared to other patch tokens. To address these artifacts, we propose an approach called Inference-Time Attention Engineering (ITAE), which manipulates attention function during inference. Specifically, we identify the artifacts by investigating one of the Query-Key-Value (QKV) patches in the multi-head attention and attenuate their corresponding attention values inside the pretrained models. ITAE shows improved clustering accuracy on multiple datasets by exhibiting more expressive features in latent space. Our findings highlight the potential of ITAE as a practical solution for reducing artifacts in pretrained ViT models and improving model performance in clustering tasks without the need for re-training or fine-tuning.
172	7e8f3fe09b52af8640d5094f617f3ba9198313d3	2410.01680	10.48550/arXiv.2410.01680	2024	PHI-S: Distribution Balancing for Label-Free Multi-Teacher Distillation	Michael Ranzinger	6	arXiv.org	20	Various visual foundation models have distinct strengths and weaknesses, both of which can be improved through heterogeneous multi-teacher knowledge distillation without labels, termed"agglomerative models."We build upon this body of work by studying the effect of the teachers' activation statistics, particularly the impact of the loss function on the resulting student model quality. We explore a standard toolkit of statistical normalization techniques to better align the different distributions and assess their effects. Further, we examine the impact on downstream teacher-matching metrics, which motivates the use of Hadamard matrices. With these matrices, we demonstrate useful properties, showing how they can be used for isotropic standardization, where each dimension of a multivariate distribution is standardized using the same scale. We call this technique"PHI Standardization"(PHI-S) and empirically demonstrate that it produces the best student model across the suite of methods studied.
173	c0df8b59bc5fd4145800360e1532eac06d45ca65	2409.07787	10.48550/arXiv.2409.07787	2024	Stable Language Model Pre-training by Reducing Embedding Variability	Woo-Jin Chung	5	Conference on Empirical Methods in Natural Language Processing	8	Stable pre-training is essential for achieving better-performing language models. However, tracking pre-training stability is impractical due to high computational costs. We study Token Embedding Variability as a simple proxy to estimate pre-training stability. We theoretically and empirically demonstrate that Multi-head Low-Rank Attention acts as a fundamental approach to reducing instability. This is supported by empirical findings on variants on GPT-2, demonstrating improved stability and lower perplexities, even at deeper layer counts.
174	66f624f30d323515447b1364a2d52228c7f85f21	2408.13152	10.48550/arXiv.2408.13152	2024	Long-Term Pre-training for Temporal Action Detection with Transformers	Jihwan Kim	3	Pattern Recognition	8	Temporal action detection (TAD) is challenging, yet fundamental for real-world video applications. Recently, DETR-based models for TAD have been prevailing thanks to their unique benefits. However, transformers demand a huge dataset, and unfortunately data scarcity in TAD causes a severe degeneration. In this paper, we identify two crucial problems from data scarcity: attention collapse and imbalanced performance. To this end, we propose a new pre-training strategy, Long-Term Pre-training (LTP), tailored for transformers. LTP has two main components: 1) class-wise synthesis, 2) long-term pretext tasks. Firstly, we synthesize long-form video features by merging video snippets of a target class and non-target classes. They are analogous to untrimmed data used in TAD, despite being created from trimmed data. In addition, we devise two types of long-term pretext tasks to learn long-term dependency. They impose long-term conditions such as finding second-to-fourth or short-duration actions. Our extensive experiments show state-of-the-art performances in DETR-based methods on ActivityNet-v1.3 and THUMOS14 by a large margin. Moreover, we demonstrate that LTP significantly relieves the data scarcity issues in TAD.
175	112a80b68542af873abc731e036d5df0bd3dfc43	2406.12719		2024	Exploring the Robustness of Language Models for Tabular Question Answering via Attention Analysis	K. Bhandari	4	Trans. Mach. Learn. Res.	10	Large Language Models (LLMs), already shown to ace various unstructured text comprehension tasks, have also remarkably been shown to tackle table (structured) comprehension tasks without specific training. Building on earlier studies of LLMs for tabular tasks, we probe how in-context learning (ICL), model scale, instruction tuning, and domain bias affect Tabular QA (TQA) robustness by testing LLMs, under diverse augmentations and perturbations, on diverse domains: Wikipedia-based $\textbf{WTQ}$, financial $\textbf{TAT-QA}$, and scientific $\textbf{SCITAB}$. Although instruction tuning and larger, newer LLMs deliver stronger, more robust TQA performance, data contamination and reliability issues, especially on $\textbf{WTQ}$, remain unresolved. Through an in-depth attention analysis, we reveal a strong correlation between perturbation-induced shifts in attention dispersion and the drops in performance, with sensitivity peaking in the model's middle layers. We highlight the need for improved interpretable methodologies to develop more reliable LLMs for table comprehension. Through an in-depth attention analysis, we reveal a strong correlation between perturbation-induced shifts in attention dispersion and performance drops, with sensitivity peaking in the model's middle layers. Based on these findings, we argue for the development of structure-aware self-attention mechanisms and domain-adaptive processing techniques to improve the transparency, generalization, and real-world reliability of LLMs on tabular data.
176	3158fc7f68b3b22545d90f87b6350c450e11026c	2406.15303	10.1007/978-3-032-04981-0_5	2024	AEM: Attention Entropy Maximization for Multiple Instance Learning Based Whole Slide Image Classification	Yunlong Zhang	7	International Conference on Medical Image Computing and Computer-Assisted Intervention	16	Multiple Instance Learning (MIL) effectively analyzes whole slide images but faces overfitting due to attention over-concentration. While existing solutions rely on complex architectural modifications or additional processing steps, we introduce Attention Entropy Maximization (AEM), a simple yet effective regularization technique. Our investigation reveals the positive correlation between attention entropy and model performance. Building on this insight, we integrate AEM regularization into the MIL framework to penalize excessive attention concentration. To address sensitivity to the AEM weight parameter, we implement Cosine Weight Annealing, reducing parameter dependency. Extensive evaluations demonstrate AEM's superior performance across diverse feature extractors, MIL frameworks, attention mechanisms, and augmentation techniques. Here is our anonymous code: https://github.com/dazhangyu123/AEM.
177	c9cecedd168c814bdf916c8ccf15457c2cc69257	2406.06811	10.48550/arXiv.2406.06811	2024	Learning Continually by Spectral Regularization	Alex Lewandowski	7	International Conference on Learning Representations	44	Loss of plasticity is a phenomenon where neural networks can become more difficult to train over the course of learning. Continual learning algorithms seek to mitigate this effect by sustaining good performance while maintaining network trainability. We develop a new technique for improving continual learning inspired by the observation that the singular values of the neural network parameters at initialization are an important factor for trainability during early phases of learning. From this perspective, we derive a new spectral regularizer for continual learning that better sustains these beneficial initialization properties throughout training. In particular, the regularizer keeps the maximum singular value of each layer close to one. Spectral regularization directly ensures that gradient diversity is maintained throughout training, which promotes continual trainability, while minimally interfering with performance in a single task. We present an experimental analysis that shows how the proposed spectral regularizer can sustain trainability and performance across a range of model architectures in continual supervised and reinforcement learning settings. Spectral regularization is less sensitive to hyperparameters while demonstrating better training in individual tasks, sustaining trainability as new tasks arrive, and achieving better generalization performance.
178	6f914dd361254e40161c2bd3916931c9d54ab4b2	2406.04267	10.48550/arXiv.2406.04267	2024	Transformers need glasses! Information over-squashing in language tasks	Federico Barbero	8	Neural Information Processing Systems	109	We study how information propagates in decoder-only Transformers, which are the architectural backbone of most existing frontier large language models (LLMs). We rely on a theoretical signal propagation analysis -- specifically, we analyse the representations of the last token in the final layer of the Transformer, as this is the representation used for next-token prediction. Our analysis reveals a representational collapse phenomenon: we prove that certain distinct sequences of inputs to the Transformer can yield arbitrarily close representations in the final token. This effect is exacerbated by the low-precision floating-point formats frequently used in modern LLMs. As a result, the model is provably unable to respond to these sequences in different ways -- leading to errors in, e.g., tasks involving counting or copying. Further, we show that decoder-only Transformer language models can lose sensitivity to specific tokens in the input, which relates to the well-known phenomenon of over-squashing in graph neural networks. We provide empirical evidence supporting our claims on contemporary LLMs. Our theory also points to simple solutions towards ameliorating these issues.
179	9654c73bb345fe4fb2dc932971f0c2e680048619	2405.19279	10.48550/arXiv.2405.19279	2024	Understanding and Minimising Outlier Features in Neural Network Training	Bobby He	5	arXiv.org	11	Outlier Features (OFs) are neurons whose activation magnitudes significantly exceed the average over a neural network's (NN) width. They are well known to emerge during standard transformer training and have the undesirable effect of hindering quantisation in afflicted models. Despite their practical importance, little is known behind why OFs emerge during training, nor how one can minimise them. Our work focuses on the above questions, first identifying several quantitative metrics, such as the kurtosis over neuron activation norms, to measure OFs. With these metrics, we study how architectural and optimisation choices influence OFs, and provide practical insights to minimise OFs during training. As highlights, we introduce a novel unnormalised transformer block, the Outlier Protected block, and present a previously unknown benefit of non-diagonal preconditioning optimisers, finding both approaches to significantly reduce OFs and improve quantisation without compromising convergence speed, at scales of up to 7B parameters. Notably, our combination of OP block and non-diagonal preconditioner (SOAP) achieves 14.87 int8 weight-and-activation perplexity (from 14.71 in standard precision), compared to 63.4 int8 perplexity (from 16.00) with a default OF-prone combination of Pre-Norm model and Adam, when quantising OPT-125m models post-training. Overall, our findings shed new light on our understanding of, our ability to prevent, and the complexity of this important aspect of NN training dynamics.
180	735c3eff45b5cabcd9efa2c0451a8fd36f840769	2405.15903	10.48550/arXiv.2405.15903	2024	UnitNorm: Rethinking Normalization for Transformers in Time Series	Nan Huang	3	arXiv.org	6	Normalization techniques are crucial for enhancing Transformer models' performance and stability in time series analysis tasks, yet traditional methods like batch and layer normalization often lead to issues such as token shift, attention shift, and sparse attention. We propose UnitNorm, a novel approach that scales input vectors by their norms and modulates attention patterns, effectively circumventing these challenges. Grounded in existing normalization frameworks, UnitNorm's effectiveness is demonstrated across diverse time series analysis tasks, including forecasting, classification, and anomaly detection, via a rigorous evaluation on 6 state-of-the-art models and 10 datasets. Notably, UnitNorm shows superior performance, especially in scenarios requiring robust attention mechanisms and contextual comprehension, evidenced by significant improvements by up to a 1.46 decrease in MSE for forecasting, and a 4.89% increase in accuracy for classification. This work not only calls for a reevaluation of normalization strategies in time series Transformers but also sets a new direction for enhancing model performance and stability. The source code is available at https://anonymous.4open.science/r/UnitNorm-5B84.
181	265b113b0fa8b6fc5192148a6f6730dddda2d2b6	2405.13985	10.48550/arXiv.2405.13985	2024	LookHere: Vision Transformers with Directed Attention Generalize and Extrapolate	A. Fuller	4	Neural Information Processing Systems	11	High-resolution images offer more information about scenes that can improve model accuracy. However, the dominant model architecture in computer vision, the vision transformer (ViT), cannot effectively leverage larger images without finetuning -- ViTs poorly extrapolate to more patches at test time, although transformers offer sequence length flexibility. We attribute this shortcoming to the current patch position encoding methods, which create a distribution shift when extrapolating. We propose a drop-in replacement for the position encoding of plain ViTs that restricts attention heads to fixed fields of view, pointed in different directions, using 2D attention masks. Our novel method, called LookHere, provides translation-equivariance, ensures attention head diversity, and limits the distribution shift that attention heads face when extrapolating. We demonstrate that LookHere improves performance on classification (avg. 1.6%), against adversarial attack (avg. 5.4%), and decreases calibration error (avg. 1.5%) -- on ImageNet without extrapolation. With extrapolation, LookHere outperforms the current SoTA position encoding method, 2D-RoPE, by 21.7% on ImageNet when trained at $224^2$ px and tested at $1024^2$ px. Additionally, we release a high-resolution test set to improve the evaluation of high-resolution image classifiers, called ImageNet-HR.
182	2b0efe3aa6c034706faf57372e0d31e1e8545f7f	2404.08634		2024	When Attention Collapses: How Degenerate Layers in LLMs Enable Smaller, Stronger Models	Sunny Sanyal	4	Trans. Mach. Learn. Res.	11	Large Language Models (LLMs) are known for their performance, but we uncover a significant structural inefficiency: a phenomenon we term attention collapse. In many pre-trained decoder-style LLMs, the attention matrices in deeper layers degenerate, collapsing to near rank-one structures. These underutilized layers, which we call lazy layers, are redundant and impair model efficiency. To address this, we introduce Inheritune, a simple yet powerful training recipe designed to build smaller, stronger language models. Inheritune initializes a compact model by inheriting the potent early layers from a larger pre-trained model and then progressively trains and expands it. Our experiments on various models, including the GPT-2 family, demonstrate that models trained with Inheritune can match or even surpass the performance of their larger counterparts, despite having significantly fewer layers. This work presents a novel path toward model compression by design, enabling the creation of compact, yet highly performant language models. Code is available at https://github.com/sanyalsunny111/LLM-Inheritune.
183	82c174ba4239b7539e4f4a87c1f89cb3e17aa6fa	2403.11052	10.48550/arXiv.2403.11052	2024	Unveiling and Mitigating Memorization in Text-to-image Diffusion Models through Cross Attention	Jie Ren	7	European Conference on Computer Vision	77	Recent advancements in text-to-image diffusion models have demonstrated their remarkable capability to generate high-quality images from textual prompts. However, increasing research indicates that these models memorize and replicate images from their training data, raising tremendous concerns about potential copyright infringement and privacy risks. In our study, we provide a novel perspective to understand this memorization phenomenon by examining its relationship with cross-attention mechanisms. We reveal that during memorization, the cross-attention tends to focus disproportionately on the embeddings of specific tokens. The diffusion model is overfitted to these token embeddings, memorizing corresponding training images. To elucidate this phenomenon, we further identify and discuss various intrinsic findings of cross-attention that contribute to memorization. Building on these insights, we introduce an innovative approach to detect and mitigate memorization in diffusion models. The advantage of our proposed method is that it will not compromise the speed of either the training or the inference processes in these models while preserving the quality of generated images. Our code is available at https://github.com/renjie3/MemAttn .
184	6e651f7953e42a6cf65c47ea17d4b47d217a9f93	2403.05490	10.48550/arXiv.2403.05490	2024	Poly-View Contrastive Learning	Amitis Shidani	6	International Conference on Learning Representations	10	Contrastive learning typically matches pairs of related views among a number of unrelated negative views. Views can be generated (e.g. by augmentations) or be observed. We investigate matching when there are more than two related views which we call poly-view tasks, and derive new representation learning objectives using information maximization and sufficient statistics. We show that with unlimited computation, one should maximize the number of related views, and with a fixed compute budget, it is beneficial to decrease the number of unique samples whilst increasing the number of views of those samples. In particular, poly-view contrastive models trained for 128 epochs with batch size 256 outperform SimCLR trained for 1024 epochs at batch size 4096 on ImageNet1k, challenging the belief that contrastive models require large batch sizes and many training epochs.
185	53a5519c724fcc547cc3e81e64b737c66abcfc56	2402.16788	10.48550/arXiv.2402.16788	2024	Why Transformers Need Adam: A Hessian Perspective	Yu-Shun Zhang	6	Neural Information Processing Systems	154	SGD performs worse than Adam by a significant margin on Transformers, but the reason remains unclear. In this work, we provide an explanation through the lens of Hessian: (i) Transformers are"heterogeneous": the Hessian spectrum across parameter blocks vary dramatically, a phenomenon we call"block heterogeneity"; (ii) Heterogeneity hampers SGD: SGD performs worse than Adam on problems with block heterogeneity. To validate (i) and (ii), we check various Transformers, CNNs, MLPs, and quadratic problems, and find that SGD can perform on par with Adam on problems without block heterogeneity, but performs worse than Adam when the heterogeneity exists. Our initial theoretical analysis indicates that SGD performs worse because it applies one single learning rate to all blocks, which cannot handle the heterogeneity among blocks. This limitation could be ameliorated if we use coordinate-wise learning rates, as designed in Adam.
186	4f5aaf3018725a4a669e4cc4ca90a125b5c3d815		10.3390/signals5010005	2024	Object Detection with Hyperparameter and Image Enhancement Optimisation for a Smart and Lean Pick-and-Place Solution	Elven Kee	4	Signals	4	Pick-and-place operations are an integral part of robotic automation and smart manufacturing. By utilizing deep learning techniques on resource-constraint embedded devices, the pick-and-place operations can be made more accurate, efficient, and sustainable, compared to the high-powered computer solution. In this study, we propose a new technique for object detection on an embedded system using SSD Mobilenet V2 FPN Lite with the optimisation of the hyperparameter and image enhancement. By increasing the Red Green Blue (RGB) saturation level of the images, we gain a 7% increase in mean Average Precision (mAP) when compared to the control group and a 20% increase in mAP when compared to the COCO 2017 validation dataset. Using a Learning Rate of 0.08 with an Edge Tensor Processing Unit (TPU), we obtain high real-time detection scores of 97%. The high detection scores are important to the control algorithm, which uses the bounding box to send a signal to the collaborative robot for pick-and-place operation.
187	7fc2323200d4cae4b3e0b316ddae0635b638417c	2402.15415		2024	Understanding Catastrophic Forgetting In LoRA via Mean-Field Attention Dynamics	Hugo Koubbi	3		7	Low-Rank Adaptation (LoRA) is the dominant parameter-efficient fine-tuning method due to its favorable compute-performance trade-off, yet it suffers from catastrophic forgetting. We study forgetting through a tractable _mean-field self-attention_ toy model, where tokens evolve as an interacting particle system and LoRA acts as a low-rank perturbation. Using tools from partial differential equations and dynamical systems, we characterize regimes suggesting a phase transition between forgetting and non-forgetting behavior. We show that one phase transition appears with respect to the norm of the perturbation, and the other with respect to the depth of the Transformers. We further bound the time-to-deviation in terms of the perturbation size and spectral quantities, and corroborate the predicted trends with experiments and exploratory analyses on real models under LoRA fine-tuning.
188	c8f5f6d981d96ebc84ba0a2665f00be3d1cfc27e	2402.10198		2024	SAMformer: Unlocking the Potential of Transformers in Time Series Forecasting with Sharpness-Aware Minimization and Channel-Wise Attention	Romain Ilbert	7	International Conference on Machine Learning	81	Transformer-based architectures achieved breakthrough performance in natural language processing and computer vision, yet they remain inferior to simpler linear baselines in multivariate long-term forecasting. To better understand this phenomenon, we start by studying a toy linear forecasting problem for which we show that transformers are incapable of converging to their true solution despite their high expressive power. We further identify the attention of transformers as being responsible for this low generalization capacity. Building upon this insight, we propose a shallow lightweight transformer model that successfully escapes bad local minima when optimized with sharpness-aware optimization. We empirically demonstrate that this result extends to all commonly used real-world multivariate time series datasets. In particular, SAMformer surpasses current state-of-the-art methods and is on par with the biggest foundation model MOIRAI while having significantly fewer parameters. The code is available at https://github.com/romilbert/samformer.
189	fefce2a7218815734cefeab724e7b69adea1df9a	2402.10202	10.48550/arXiv.2402.10202	2024	Bridging Associative Memory and Probabilistic Modeling	Rylan Schaeffer	12	arXiv.org	7	Associative memory and probabilistic modeling are two fundamental topics in artificial intelligence. The first studies recurrent neural networks designed to denoise, complete and retrieve data, whereas the second studies learning and sampling from probability distributions. Based on the observation that associative memory's energy functions can be seen as probabilistic modeling's negative log likelihoods, we build a bridge between the two that enables useful flow of ideas in both directions. We showcase four examples: First, we propose new energy-based models that flexibly adapt their energy functions to new in-context datasets, an approach we term \textit{in-context learning of energy functions}. Second, we propose two new associative memory models: one that dynamically creates new memories as necessitated by the training data using Bayesian nonparametrics, and another that explicitly computes proportional memory assignments using the evidence lower bound. Third, using tools from associative memory, we analytically and numerically characterize the memory capacity of Gaussian kernel density estimators, a widespread tool in probababilistic modeling. Fourth, we study a widespread implementation choice in transformers -- normalization followed by self attention -- to show it performs clustering on the hypersphere. Altogether, this work urges further exchange of useful ideas between these two continents of artificial intelligence.
190	efb1610823d5ae1807540f04a8af2485b637dcc0	2402.02634	10.48550/arXiv.2402.02634	2024	Key-Graph Transformer for Image Restoration	Bin Ren	8	arXiv.org	3	While it is crucial to capture global information for effective image restoration (IR), integrating such cues into transformer-based methods becomes computationally expensive, especially with high input resolution. Furthermore, the self-attention mechanism in transformers is prone to considering unnecessary global cues from unrelated objects or regions, introducing computational inefficiencies. In response to these challenges, we introduce the Key-Graph Transformer (KGT) in this paper. Specifically, KGT views patch features as graph nodes. The proposed Key-Graph Constructor efficiently forms a sparse yet representative Key-Graph by selectively connecting essential nodes instead of all the nodes. Then the proposed Key-Graph Attention is conducted under the guidance of the Key-Graph only among selected nodes with linear computational complexity within each window. Extensive experiments across 6 IR tasks confirm the proposed KGT's state-of-the-art performance, showcasing advancements both quantitatively and qualitatively.
191	b6dd8671099ed834850290530a6dc347303adfbf	2402.02098	10.48550/arXiv.2402.02098	2024	Self-attention Networks Localize When QK-eigenspectrum Concentrates	Han Bao	3	International Conference on Machine Learning	20	The self-attention mechanism prevails in modern machine learning. It has an interesting functionality of adaptively selecting tokens from an input sequence by modulating the degree of attention localization, which many researchers speculate is the basis of the powerful model performance but complicates the underlying mechanism of the learning dynamics. In recent years, mainly two arguments have connected attention localization to the model performances. One is the rank collapse, where the embedded tokens by a self-attention block become very similar across different tokens, leading to a less expressive network. The other is the entropy collapse, where the attention probability approaches non-uniform and entails low entropy, making the learning dynamics more likely to be trapped in plateaus. These two failure modes may apparently contradict each other because the rank and entropy collapses are relevant to uniform and non-uniform attention, respectively. To this end, we characterize the notion of attention localization by the eigenspectrum of query-key parameter matrices and reveal that a small eigenspectrum variance leads attention to be localized. Interestingly, the small eigenspectrum variance prevents both rank and entropy collapse, leading to better model expressivity and trainability.
192	603242e88244aff1135ce6cb3baf53d2e95ed427	2401.09865	10.48550/arXiv.2401.09865	2024	Improving fine-grained understanding in image-text pre-training	Ioana Bica	11	International Conference on Machine Learning	64	We introduce SPARse Fine-grained Contrastive Alignment (SPARC), a simple method for pretraining more fine-grained multimodal representations from image-text pairs. Given that multiple image patches often correspond to single words, we propose to learn a grouping of image patches for every token in the caption. To achieve this, we use a sparse similarity metric between image patches and language tokens and compute for each token a language-grouped vision embedding as the weighted average of patches. The token and language-grouped vision embeddings are then contrasted through a fine-grained sequence-wise loss that only depends on individual samples and does not require other batch samples as negatives. This enables more detailed information to be learned in a computationally inexpensive manner. SPARC combines this fine-grained loss with a contrastive loss between global image and text embeddings to learn representations that simultaneously encode global and local information. We thoroughly evaluate our proposed method and show improved performance over competing approaches both on image-level tasks relying on coarse-grained information, e.g. classification, as well as region-level tasks relying on fine-grained information, e.g. retrieval, object detection, and segmentation. Moreover, SPARC improves model faithfulness and captioning in foundational vision-language models.
193	c43f5c366fff23b38f179779c4c05f6d15c09dcc	2401.04301	10.48550/arXiv.2401.04301	2024	Setting the Record Straight on Transformer Oversmoothing	G. Dovonon	3	Trans. Mach. Learn. Res.	23	Transformer-based models have recently become wildly successful across a diverse set of domains. At the same time, recent work has shown empirically and theoretically that Transformers are inherently limited. Specifically, they argue that as model depth increases, Transformers oversmooth, i.e., inputs become more and more similar. A natural question is: How can Transformers achieve these successes given this shortcoming? In this work we test these observations empirically and theoretically and uncover a number of surprising findings. We find that there are cases where feature similarity increases but, contrary to prior results, this is not inevitable, even for existing pre-trained models. Theoretically, we show that smoothing behavior depends on the eigenspectrum of the value and projection weights. We verify this empirically and observe that the sign of layer normalization weights can influence this effect. Our analysis reveals a simple way to parameterize the weights of the Transformer update equations to influence smoothing behavior. We hope that our findings give ML researchers and practitioners additional insight into how to develop future Transformer-based models.
194	c431b1a922385830d4c934a5679edfb9cedc600b	2312.16903	10.48550/arXiv.2312.16903	2023	Spike No More: Stabilizing the Pre-training of Large Language Models	Sho Takase	4	arXiv.org	56	Loss spikes often occur during pre-training of large language models. The spikes degrade the performance of large language models and sometimes ruin the pre-training. Since the pre-training needs a vast computational budget, we should avoid such spikes. Based on the assumption that the loss spike is caused by the sudden growth of the gradient norm, we explore factors to keep the gradient norm small through an analysis of the spectral norms of the Jacobian matrices for the sub-layers. Our findings suggest that stabilizing the pre-training process requires two conditions: small sub-layers and large shortcut. We conduct various experiments to empirically verify our theoretical analyses. Experimental results demonstrate that methods satisfying the conditions effectively prevent loss spikes during pre-training.
195	45f58654538d020f606f4ae26f996df4204c199d	2312.17372	10.48550/arXiv.2312.17372	2023	Beyond PID Controllers: PPO with Neuralized PID Policy for Proton Beam Intensity Control in Mu2e	Chenwei Xu	24	arXiv.org	3	We introduce a novel Proximal Policy Optimization (PPO) algorithm aimed at addressing the challenge of maintaining a uniform proton beam intensity delivery in the Muon to Electron Conversion Experiment (Mu2e) at Fermi National Accelerator Laboratory (Fermilab). Our primary objective is to regulate the spill process to ensure a consistent intensity profile, with the ultimate goal of creating an automated controller capable of providing real-time feedback and calibration of the Spill Regulation System (SRS) parameters on a millisecond timescale. We treat the Mu2e accelerator system as a Markov Decision Process suitable for Reinforcement Learning (RL), utilizing PPO to reduce bias and enhance training stability. A key innovation in our approach is the integration of a neuralized Proportional-Integral-Derivative (PID) controller into the policy function, resulting in a significant improvement in the Spill Duty Factor (SDF) by 13.6%, surpassing the performance of the current PID controller baseline by an additional 1.6%. This paper presents the preliminary offline results based on a differentiable simulator of the Mu2e accelerator. It paves the groundwork for real-time implementations and applications, representing a crucial step towards automated proton beam intensity control for the Mu2e experiment.
196	78d5723277bde5d37c2b6230aba83f749651ab10	2312.06182	10.48550/arXiv.2312.06182	2023	Why "classic" Transformers are shallow and how to make them go deep	Yueyao Yu	2	arXiv.org	1	Since its introduction in 2017, Transformer has emerged as the leading neural network architecture, catalyzing revolutionary advancements in many AI disciplines. The key innovation in Transformer is a Self-Attention (SA) mechanism designed to capture contextual information. However, extending the original Transformer design to models of greater depth has proven exceedingly challenging, if not impossible. Even though various modifications have been proposed in order to stack more layers of SA mechanism into deeper models, a full understanding of this depth problem remains lacking. In this paper, we conduct a comprehensive investigation, both theoretically and empirically, to substantiate the claim that the depth problem is caused by \emph{token similarity escalation}; that is, tokens grow increasingly alike after repeated applications of the SA mechanism. Our analysis reveals that, driven by the invariant leading eigenspace and large spectral gaps of attention matrices, token similarity provably escalates at a linear rate. Based on the gained insight, we propose a new strategy of surgically removing excessive similarity in contrast to the existing approach of diminishing the SA mechanism explicitly or implicitly (such as in pre-norm transformers). Preliminary experimental results confirm the effectiveness of the proposed strategy in small-scale post-norm Transformer models.
197	2fd95b3c79e0c39bdd2611bdc3cc369c6c73d8b1	2312.06709	10.1109/CVPR52733.2024.01187	2023	AM-RADIO: Agglomerative Vision Foundation Model Reduce All Domains Into One	Michael Ranzinger	4	Computer Vision and Pattern Recognition	221	A handful of visual foundation models (VFMs) have recently emerged as the backbones for numerous downstream tasks. VFMs like CLIP, DINOv2, SAM are trained with distinct objectives, exhibiting unique characteristics for various downstream tasks. We find that despite their conceptual differences, these models can be effectively merged into a unified model through multi-teacher distillation. We name this approach AM-RADIO (Agglomerative Model - Reduce All Domains Into One). This integrative approach not only surpasses the performance of individual teacher models but also amalgamates their distinctive features, such as zero-shot vision-language comprehension, detailed pixel-level understanding, and open vocabulary segmentation capabilities. Additionally, in pursuit of the most hardware-efficient backbone, we evaluated numerous architectures in our multi-teacher distillation pipeline using the same training recipe. This led to the development of a novel architecture (E-RADIO) that exceeds the performance of its predecessors and is at least 6x faster than the teacher models at matched resolution. Our comprehensive benchmarking process covers downstream tasks including ImageNet classification, semantic segmentation linear probing, COCO object detection and integration into LLaVa-1.5. Code: https://github.com/NVlabs/RADIO.
198	a8404aa3b7eb9799be52a2a8789bea3733da8950	2312.04234	10.48550/arXiv.2312.04234	2023	Graph Convolutions Enrich the Self-Attention in Transformers!	Jeongwhan Choi	7	Neural Information Processing Systems	24	Transformers, renowned for their self-attention mechanism, have achieved state-of-the-art performance across various tasks in natural language processing, computer vision, time-series modeling, etc. However, one of the challenges with deep Transformer models is the oversmoothing problem, where representations across layers converge to indistinguishable values, leading to significant performance degradation. We interpret the original self-attention as a simple graph filter and redesign it from a graph signal processing (GSP) perspective. We propose a graph-filter-based self-attention (GFSA) to learn a general yet effective one, whose complexity, however, is slightly larger than that of the original self-attention mechanism. We demonstrate that GFSA improves the performance of Transformers in various fields, including computer vision, natural language processing, graph-level tasks, speech recognition, and code classification.
199	5fdbd2f2fb3b6e15d68590e4934b9316128cc6b2		10.1016/j.compbiomed.2023.107866	2023	SaB-Net: Self-attention backward network for gastric tumor segmentation in CT images	Junjie He	8	Comput. Biol. Medicine	19	Gastric cancer is a significant contributor to cancer-related fatalities globally. The automated segmentation of gastric tumors has the potential to analyze the medical condition of patients and enhance the likelihood of surgical treatment success. However, the development of an automatic solution is challenged by the heterogeneous intensity distribution of gastric tumors in computed tomography (CT) images, the low-intensity contrast between organs, and the high variability in the stomach shapes and gastric tumors in different patients. To address these challenges, we propose a self-attention backward network (SaB-Net) for gastric tumor segmentation (GTS) in CT images by introducing a self-attention backward layer (SaB-Layer) to feed the self-attention information learned at the deep layer back to the shallow layers. The SaB-Layer efficiently extracts tumor information from CT images and integrates the information into the network, thereby enhancing the network's tumor segmentation ability. We employed datasets from two centers, one for model training and testing and the other for external validation. The model achieved dice scores of 0.8456 on the test set and 0.8068 on the external verification set. Moreover, we validated the model's transfer learning ability on a publicly available liver cancer dataset, achieving results comparable to state-of-the-art liver cancer segmentation models recently developed. SaB-Net has strong potential for assisting in the clinical diagnosis of and therapy for gastric cancer. Our implementation is available at https://github.com/TyrionJ/SaB-Net.
200	5e2cada56906540d31a73026a1e4d78cb8faf971	2310.12956	10.48550/arXiv.2310.12956	2023	Eureka-Moments in Transformers: Multi-Step Tasks Reveal Softmax Induced Optimization Problems	David T. Hoffmann	6	International Conference on Machine Learning	16	In this work, we study rapid improvements of the training loss in transformers when being confronted with multi-step decision tasks. We found that transformers struggle to learn the intermediate task and both training and validation loss saturate for hundreds of epochs. When transformers finally learn the intermediate task, they do this rapidly and unexpectedly. We call these abrupt improvements Eureka-moments, since the transformer appears to suddenly learn a previously incomprehensible concept. We designed synthetic tasks to study the problem in detail, but the leaps in performance can be observed also for language modeling and in-context learning (ICL). We suspect that these abrupt transitions are caused by the multi-step nature of these tasks. Indeed, we find connections and show that ways to improve on the synthetic multi-step tasks can be used to improve the training of language modeling and ICL. Using the synthetic data we trace the problem back to the Softmax function in the self-attention block of transformers and show ways to alleviate the problem. These fixes reduce the required number of training steps, lead to higher likelihood to learn the intermediate task, to higher final accuracy and training becomes more robust to hyper-parameters.
201	f5789596531fad358c3166fdb5bd72d8e661c32c	2309.14322	10.48550/arXiv.2309.14322	2023	Small-scale proxies for large-scale Transformer training instabilities	Mitchell Wortsman	16	International Conference on Learning Representations	210	Teams that have trained large Transformer-based models have reported training instabilities at large scale that did not appear when training with the same hyperparameters at smaller scales. Although the causes of such instabilities are of scientific interest, the amount of resources required to reproduce them has made investigation difficult. In this work, we seek ways to reproduce and study training stability and instability at smaller scales. First, we focus on two sources of training instability described in previous work: the growth of logits in attention layers (Dehghani et al., 2023) and divergence of the output logits from the log probabilities (Chowdhery et al., 2022). By measuring the relationship between learning rate and loss across scales, we show that these instabilities also appear in small models when training at high learning rates, and that mitigations previously employed at large scales are equally effective in this regime. This prompts us to investigate the extent to which other known optimizer and model interventions influence the sensitivity of the final loss to changes in the learning rate. To this end, we study methods such as warm-up, weight decay, and the $\mu$Param (Yang et al., 2022), and combine techniques to train small models that achieve similar losses across orders of magnitude of learning rate variation. Finally, to conclude our exploration we study two cases where instabilities can be predicted before they emerge by examining the scaling behavior of model activation and gradient norms.
202	c1738f21ea2460e1015d590906a4f43e155f60c8	2306.17759	10.48550/arXiv.2306.17759	2023	The Shaped Transformer: Attention Models in the Infinite Depth-and-Width Limit	Lorenzo Noci	7	Neural Information Processing Systems	60	In deep learning theory, the covariance matrix of the representations serves as a proxy to examine the network's trainability. Motivated by the success of Transformers, we study the covariance matrix of a modified Softmax-based attention model with skip connections in the proportional limit of infinite-depth-and-width. We show that at initialization the limiting distribution can be described by a stochastic differential equation (SDE) indexed by the depth-to-width ratio. To achieve a well-defined stochastic limit, the Transformer's attention mechanism is modified by centering the Softmax output at identity, and scaling the Softmax logits by a width-dependent temperature parameter. We examine the stability of the network through the corresponding SDE, showing how the scale of both the drift and diffusion can be elegantly controlled with the aid of residual connections. The existence of a stable SDE implies that the covariance structure is well-behaved, even for very large depth and width, thus preventing the notorious issues of rank degeneracy in deep attention models. Finally, we show, through simulations, that the SDE provides a surprisingly good description of the corresponding finite-size model. We coin the name shaped Transformer for these architectural modifications.
203	b3c64a046d6cfc0f55a2aebc5176bbab69a7e021	2304.13013	10.48550/arXiv.2304.13013	2023	Stable and low-precision training for large-scale vision-language models	Mitchell Wortsman	6	Neural Information Processing Systems	106	We introduce new methods for 1) accelerating and 2) stabilizing training for large language-vision models. 1) For acceleration, we introduce SwitchBack, a linear layer for int8 quantized training which provides a speed-up of 13-25% while matching the performance of bfloat16 training within 0.1 percentage points for the 1B parameter CLIP ViT-Huge -- the largest int8 training to date. Our main focus is int8 as GPU support for float8 is rare, though we also analyze float8 training through simulation. While SwitchBack proves effective for float8, we show that standard techniques are also successful if the network is trained and initialized so that large feature magnitudes are discouraged, which we accomplish via layer-scale initialized with zeros. 2) For stability, we analyze loss spikes and find they consistently occur 1-8 iterations after the squared gradients become under-estimated by their AdamW second moment estimator. As a result, we recommend an AdamW-Adafactor hybrid which avoids loss spikes when training a CLIP ViT-Huge model and outperforms gradient clipping at the scales we test.
204	618795552db11785d39976732d9ae5629d3932ff		10.48550/arXiv.2604.17324	2026	SigGate-GT: Taming Over-Smoothing in Graph Transformers via Sigmoid-Gated Attention	Dong-Xing Guo	3	arXiv.org	1
205	96343c1f4c33868288f197ef7bbb059040f7f1b5		10.1109/ACCESS.2026.3717642	2026	HTAC-Net: Hierarchical Temporal Acoustic Composition Estimation and Biological-Component Detection in Controlled Underwater Soundscapes	Vaka Suresh	2	IEEE Access	1	Underwater acoustic recordings often contain overlapping biological, anthropogenic, and geophysical sources, whereas many existing methods formulate acoustic analysis as binary detection or multi-class classification. This paper proposes HTAC-Net, a Hierarchical Temporal Acoustic Composition Network for continuous bio–anthro–geo composition estimation and supervised biological-component detection. HTAC-Net combines eight interpretable acoustic descriptors, multi-scale dilated temporal convolution, causal self-attention, a Dirichlet composition head, and a supervised detection head. The framework is evaluated using a controlled real-source synthetic-mixture benchmark in which real underwater recordings provide the source material and known Dirichlet-controlled mixing coefficients define the composition targets. The implementation focuses on the <inline-formula> <tex-math notation="LaTeX">$0-500~\mathrm {Hz}$ </tex-math></inline-formula> low-frequency band. Under a scenario-stratified full-window split-boundary leakage-control protocol with <inline-formula> <tex-math notation="LaTeX">$K=20$ </tex-math></inline-formula>, five HTAC-Net runs achieved a final-step composition mean absolute error of <inline-formula> <tex-math notation="LaTeX">$0.1168\pm 0.0012$ </tex-math></inline-formula>, a biological-detection F1-score of <inline-formula> <tex-math notation="LaTeX">$0.8701\pm 0.0058$ </tex-math></inline-formula>, an AUC-ROC of <inline-formula> <tex-math notation="LaTeX">$0.8974\pm 0.0054$ </tex-math></inline-formula>, and an AUC-PR of <inline-formula> <tex-math notation="LaTeX">$0.9602\pm 0.0020$ </tex-math></inline-formula>. A common-five-seed comparison showed statistically comparable composition-estimation accuracy for HTAC-Net and BiLSTM, with MAE values of <inline-formula> <tex-math notation="LaTeX">$0.1168\pm 0.0012$ </tex-math></inline-formula> and <inline-formula> <tex-math notation="LaTeX">$0.1166\pm 0.0008$ </tex-math></inline-formula>, respectively (paired Wilcoxon <inline-formula> <tex-math notation="LaTeX">$p=0.625$ </tex-math></inline-formula>). External weak-label analysis on 115 labeled NOAA-NRS, ONC, and Orcasound clips yielded dominant-class agreement of 0.6174, macro-F1 of 0.6123, and balanced accuracy of 0.6936, providing additional evidence of cross-domain plausibility. HTAC-Net therefore provides an integrated framework for controlled acoustic composition estimation, supervised biological-component detection, and concentration-derived diagnostic reporting. An exploratory acoustic-scene suitability score is additionally presented as a basis for future communication-layer evaluation. The reported results characterize controlled real-source synthetic-mixture benchmark performance and external weak-label plausibility.
206	5683ce0a1bb309b85c917c87aaa8748316019a12		10.48550/arXiv.2502.16570	2025	Entropy-Lens: The Information Signature of Transformer Computations	Riccardo Ali	4	arXiv.org	22
207	07d64f525e478d095bb4b38ec72c1d747fbfebb5			2025	Intervening Anchor Token: Decoding Strategy in Alleviating Hallucinations for MLLMs	Feilong Tang	6	International Conference on Learning Representations	43
208	34ac27b6b3c21b7fa1f681731d3c9ee3b5369e6b		10.1016/j.epsr.2025.111863	2025	Synchronized multi-node energy consumption prediction in complex manufacturing systems using Attention-Optimized Spatio-Temporal Deep Learning Network	Mingdong Han	2	Electric power systems research	1
209	def3d3384dcf367348835ad1e3573edfeb03d728		10.1109/TGRS.2025.3601517	2025	GSFANet: Global Spatial–Frequency Attention Network for Infrared Small Target Detection	Chui-Yi Deng	6	IEEE Transactions on Geoscience and Remote Sensing	40	Infrared small target detection (IRSTD) has progressed significantly in spatial-domain learning. However, single-frame images’ limited spatial semantics impair discrimination between targets and similar noise while complicating integrity detection of large-scale target. To address these, we propose global spatial–frequency attention network (GSFANet), which enhances the distribution difference between targets and noise from a frequency-domain perspective while preserving spatial information integrity. The core innovations consist of three modules: 1) parametric wavelet downsampling (PWD), preserving small target details during frequency refinement to prevent feature fragmentation; 2) hierarchical gated kernel attention (HGKA), capturing cross-level frequency relationships through cross-channel kernel attention ( $\text {C}^{2}\text {K}$ ) and maintaining spatial coherence via cross-spatial gate attention (CSG), effectively bridging semantic gaps across layers; and 3) adaptive frequency-decoupled fusion (AdaFD), dynamically fusing target-associated frequency components while suppressing noise. We further develop AdaFL Loss to balance multiscale target gradients and stabilize training. Experiments on three benchmark datasets demonstrate GSFANet’s superior detection performance and enhanced segmentation robustness in complex scenarios compared with state-of-the-art (SOTA) methods. Our code will be made public at https://github.com/dengfa02/GSFANet_IRSTD
210	870e842867e31920088bcda5965f0a5371ea0b5c			2025	Clique Number Estimation via Differentiable Functions of Adjacency Matrix Permutations	Indradyumna Roy	4	International Conference on Learning Representations	1
211	c8c75c4dfe7a47cc2096b26a3d70d4d14c2d1a34		10.48550/arXiv.2502.17055	2025	Stable-SPAM: How to Train in 4-Bit More Stably than 16-Bit Adam	Tian-Jin Huang	11	arXiv.org	8
212	3ec3bce143f97b5e445f2355f86b1db7978a1531		10.52202/085713-0419	2025	Energy Landscape-Aware Vision Transformers: Layerwise Dynamics and Adaptive Task-Specific Training via Hopfield States	Runze Xia	2	Neural Information Processing Systems	1	Recent advances in Vision Transformers (ViTs) have shown remarkable performance across vision tasks, yet their deep, uniform layer structure introduces significant computational overhead. In this work, we explore the emergent dynamics of ViT layers through the lens of energy-based memory systems, drawing a connection between self-attention and modern Hopfield networks. We introduce a novel metric—Layer Instability Index (LII)—derived from the operational softmax mode and its variability, to quantify the metastability of each Transformer layer over time. Our analysis reveals that certain layers exhibit consistent convergence to attractor-like states, suggesting functional specialisation and early stabilisation. Leveraging this insight, we propose an adaptive training framework that dynamically freezes or skips stable layers based on their energy landscape behavior. Our method reduces training costs while maintaining or improving accuracy. Extensive experiments on ViT-S/B/L on CUB-200-2011, CIFAR-10/100, Food-101, Stanford Dogs, and Beans demonstrate the generality and efficiency of our approach. This work provides new theoretical and practical perspectives for energy-aware optimisation of deep Transformer models.
213	9d017afe9fc342493a602ead6eb1f969836653d6		10.48550/arXiv.2402.10198	2024	Unlocking the Potential of Transformers in Time Series Forecasting with Sharpness-Aware Minimization and Channel-Wise Attention	Romain Ilbert	7	arXiv.org	19
214	498b038116ba22a205b30e3afd520045dd297862			2024	PINE: Efficient Verification of a Euclidean Norm Bound of a Secret-Shared Vector	G. Rothblum	4	USENIX Security Symposium	12
215	c373cb2cb16b65022de1d9447afa6d0f0aefb8e2			2023	Overleaf Example	Jake Grigsby	3		49
216	84709b0cff5cc625f5caeb2dc501990b811e9da6		10.48550/arXiv.2306.11986	2023	Addressing the Rank Degeneration in Sequential Recommendation via Singular Spectrum Smoothing	Ziwei Fan	4	arXiv.org	3
217	217ba497bef71f14fca31bede5233ff1f759a1c4		10.48550/arXiv.2310.00098	2023	Federated Learning with Differential Privacy for End-to-End Speech Recognition	Martin Pelikan	6	arXiv.org	16
218	75a1fd4424d7a211e332fa45fbe2554fd336beaa				From Mode Collapse to State-of-the-Art: Engineering Robust Vision-Based 3D Hand-Object Manipulation Understanding	B. Dong	2		0
219	88fc6f475d9ea9cae96972e0afb326ff6b4f826a				Variance Sensitivity Induces Attention Entropy Collapse in Transformers	J. Hong	2		0
220	9b2acdf1888c203c7617669c719fa5ba4eea1e31				Federated Learning for Speech Recognition: Revisiting Current Trends Towards Large-Scale ASR	Sheikh Shams Azam	7		10
221	d891fde1dd39eaf083d899ceffa51a87d8705b6e				Visual Geometry Grounded Transformer		0		35
222	7a783a3a2c32dafeffb6005445a2424b67abc047				Why “Classic” Transformers Are Shallow and A Depth-Enabling Technique	Yueyao Yu	2		1