Cormac Shannon
95b5857189
Six seeds snowballed (Semantic Scholar) plus logged keyword searches (OpenAlex, arXiv): 2,127 unique records, 480 title/abstract screened. AI first-pass suggestions are in their own column; decisions are pending. Claude-Session: https://claude.ai/code/session_01S8KN7NuUCm8nXLmy6eDuVv
59 KiB
59 KiB
| 1 | s2_id | arxiv | doi | year | title | first_author | n_authors | venue | citations | abstract |
|---|---|---|---|---|---|---|---|---|---|---|
| 2 | 37c1a63152c219ee8f630ecdea18f782074389e9 | 2212.06138 | 10.48550/arXiv.2212.06138 | 2022 | CLIP Itself is a Strong Fine-tuner: Achieving 85.7% and 88.0% Top-1 Accuracy with ViT-B and ViT-L on ImageNet | Xiaoyi Dong | 10 | arXiv.org | 58 | Recent studies have shown that CLIP has achieved remarkable success in performing zero-shot inference while its fine-tuning performance is not satisfactory. In this paper, we identify that fine-tuning performance is significantly impacted by hyper-parameter choices. We examine various key hyper-parameters and empirically evaluate their impact in fine-tuning CLIP for classification tasks through a comprehensive study. We find that the fine-tuning performance of CLIP is substantially underestimated. Equipped with hyper-parameter refinement, we demonstrate CLIP itself is better or at least competitive in fine-tuning compared with large-scale supervised pre-training approaches or latest works that use CLIP as prediction targets in Masked Image Modeling. Specifically, CLIP ViT-Base/16 and CLIP ViT-Large/14 can achieve 85.7%,88.0% finetuning Top-1 accuracy on the ImageNet-1K dataset . These observations challenge the conventional conclusion that CLIP is not suitable for fine-tuning, and motivate us to rethink recently proposed improvements based on CLIP. We will release our code publicly at \url{https://github.com/LightDXY/FT-CLIP}. |
| 3 | 84f6ab620eb7b112e5b2ca64b305970894e679c1 | 2207.14484 | 10.48550/arXiv.2207.14484 | 2022 | Adaptive Gradient Methods at the Edge of Stability | Jeremy M. Cohen | 11 | arXiv.org | 105 | Very little is known about the training dynamics of adaptive gradient methods like Adam in deep learning. In this paper, we shed light on the behavior of these algorithms in the full-batch and sufficiently large batch settings. Specifically, we empirically demonstrate that during full-batch training, the maximum eigenvalue of the preconditioned Hessian typically equilibrates at a certain numerical value -- the stability threshold of a gradient descent algorithm. For Adam with step size $\eta$ and $\beta_1 = 0.9$, this stability threshold is $38/\eta$. Similar effects occur during minibatch training, especially as the batch size grows. Yet, even though adaptive methods train at the ``Adaptive Edge of Stability'' (AEoS), their behavior in this regime differs in a significant way from that of non-adaptive methods at the EoS. Whereas non-adaptive algorithms at the EoS are blocked from entering high-curvature regions of the loss landscape, adaptive gradient methods at the AEoS can keep advancing into high-curvature regions, while adapting the preconditioner to compensate. Our findings can serve as a foundation for the community's future understanding of adaptive gradient methods in deep learning. |
| 4 | 5eeb80dc67590422db64ca95ec0aded24799cfb6 | 2206.03126 | 10.48550/arXiv.2206.03126 | 2022 | Signal Propagation in Transformers: Theoretical Perspectives and the Role of Rank Collapse | Lorenzo Noci | 6 | Neural Information Processing Systems | 171 | Transformers have achieved remarkable success in several domains, ranging from natural language processing to computer vision. Nevertheless, it has been recently shown that stacking self-attention layers - the distinctive architectural component of Transformers - can result in rank collapse of the tokens' representations at initialization. The question of if and how rank collapse affects training is still largely unanswered, and its investigation is necessary for a more comprehensive understanding of this architecture. In this work, we shed new light on the causes and the effects of this phenomenon. First, we show that rank collapse of the tokens' representations hinders training by causing the gradients of the queries and keys to vanish at initialization. Furthermore, we provide a thorough description of the origin of rank collapse and discuss how to prevent it via an appropriate depth-dependent scaling of the residual branches. Finally, our analysis unveils that specific architectural hyperparameters affect the gradients of queries and values differently, leading to disproportionate gradient norms. This suggests an explanation for the widespread use of adaptive methods for Transformers' optimization. |
| 5 | d2f63b56fc6bc373f5c023454c2b253326962865 | 2204.07118 | 10.48550/arXiv.2204.07118 | 2022 | DeiT III: Revenge of the ViT | Hugo Touvron | 3 | European Conference on Computer Vision | 729 | A Vision Transformer (ViT) is a simple neural architecture amenable to serve several computer vision tasks. It has limited built-in architectural priors, in contrast to more recent architectures that incorporate priors either about the input data or of specific tasks. Recent works show that ViTs benefit from self-supervised pre-training, in particular BerT-like pre-training like BeiT. In this paper, we revisit the supervised training of ViTs. Our procedure builds upon and simplifies a recipe introduced for training ResNet-50. It includes a new simple data-augmentation procedure with only 3 augmentations, closer to the practice in self-supervised learning. Our evaluations on Image classification (ImageNet-1k with and without pre-training on ImageNet-21k), transfer learning and semantic segmentation show that our procedure outperforms by a large margin previous fully supervised training recipes for ViT. It also reveals that the performance of our ViT trained with supervision is comparable to that of more recent architectures. Our results could serve as better baselines for recent self-supervised approaches demonstrated on ViT. |
| 6 | a6e231db70a6774abc995afc0201671e0dab4d3a | 2204.07141 | 10.48550/arXiv.2204.07141 | 2022 | Masked Siamese Networks for Label-Efficient Learning | Mahmoud Assran | 9 | European Conference on Computer Vision | 456 | We propose Masked Siamese Networks (MSN), a self-supervised learning framework for learning image representations. Our approach matches the representation of an image view containing randomly masked patches to the representation of the original unmasked image. This self-supervised pre-training strategy is particularly scalable when applied to Vision Transformers since only the unmasked patches are processed by the network. As a result, MSNs improve the scalability of joint-embedding architectures, while producing representations of a high semantic level that perform competitively on low-shot image classification. For instance, on ImageNet-1K, with only 5,000 annotated images, our base MSN model achieves 72.4% top-1 accuracy, and with 1% of ImageNet-1K labels, we achieve 75.7% top-1 accuracy, setting a new state-of-the-art for self-supervised learning on this benchmark. Our code is publicly available. |
| 7 | 0b0d7d87c58d41b92d907347b778032be5966f60 | 2203.03466 | 10.48550/arXiv.2203.03466 | 2022 | Tensor Programs V: Tuning Large Neural Networks via Zero-Shot Hyperparameter Transfer | Greg Yang | 10 | arXiv.org | 340 | Hyperparameter (HP) tuning in deep learning is an expensive process, prohibitively so for neural networks (NNs) with billions of parameters. We show that, in the recently discovered Maximal Update Parametrization (muP), many optimal HPs remain stable even as model size changes. This leads to a new HP tuning paradigm we call muTransfer: parametrize the target model in muP, tune the HP indirectly on a smaller model, and zero-shot transfer them to the full-sized model, i.e., without directly tuning the latter at all. We verify muTransfer on Transformer and ResNet. For example, 1) by transferring pretraining HPs from a model of 13M parameters, we outperform published numbers of BERT-large (350M parameters), with a total tuning cost equivalent to pretraining BERT-large once; 2) by transferring from 40M parameters, we outperform published numbers of the 6.7B GPT-3 model, with tuning cost only 7% of total pretraining cost. A Pytorch implementation of our technique can be found at github.com/microsoft/mup and installable via `pip install mup`. |
| 8 | e0995bad59c8638ea8c319bb7220c0f0b1ed5dca | 2203.00555 | 10.1109/TPAMI.2024.3386927 | 2022 | DeepNet: Scaling Transformers to 1,000 Layers | Hongyu Wang | 6 | IEEE Transactions on Pattern Analysis and Machine Intelligence | 284 | In this paper, we propose a simple yet effective method to stabilize extremely deep Transformers. Specifically, we introduce a new normalization function (DeepNorm) to modify the residual connection in Transformer, accompanying with theoretically derived initialization. In-depth theoretical analysis shows that model updates can be bounded in a stable way. The proposed method combines the best of two worlds, i.e., good performance of Post-LN and stable training of Pre-LN, making DeepNorm a preferred alternative. We successfully scale Transformers up to 1,000 layers (i.e., 2,500 attention and feed-forward network sublayers) without difficulty, which is one order of magnitude deeper than previous deep Transformers. Extensive experiments demonstrate that DeepNet has superior performance across various benchmarks, including machine translation, language modeling (i.e., BERT, GPT) and vision pre-training (i.e., BEiT). Remarkably, on a multilingual benchmark with 7,482 translation directions, our 200-layer model with 3.2B parameters significantly outperforms the 48-layer state-of-the-art model with 12B parameters by 5 BLEU points, which indicates a promising scaling direction. |
| 9 | 00754af9cd0eaf613a99b13fb19e422b09beeee4 | 2202.00980 | 2022 | Robust Training of Neural Networks using Scale Invariant Architectures | Zhiyuan Li | 5 | International Conference on Machine Learning | 37 | In contrast to SGD, adaptive gradient methods like Adam allow robust training of modern deep networks, especially large language models. However, the use of adaptivity not only comes at the cost of extra memory but also raises the fundamental question: can non-adaptive methods like SGD enjoy similar benefits? In this paper, we provide an affirmative answer to this question by proposing to achieve both robust and memory-efficient training via the following general recipe: (1) modify the architecture and make it scale invariant, i.e. the scale of parameter doesn't affect the output of the network, (2) train with SGD and weight decay, and optionally (3) clip the global gradient norm proportional to weight norm multiplied by $\sqrt{\tfrac{2\lambda}{\eta}}$, where $\eta$ is learning rate and $\lambda$ is weight decay. We show that this general approach is robust to rescaling of parameter and loss by proving that its convergence only depends logarithmically on the scale of initialization and loss, whereas the standard SGD might not even converge for many initializations. Following our recipe, we design a scale invariant version of BERT, called SIBERT, which when trained simply by vanilla SGD achieves performance comparable to BERT trained by adaptive methods like Adam on downstream tasks. | |
| 10 | 6351ebb4a3287f5f3e1273464b3b91e5df5a16d7 | 2111.06377 | 10.1109/CVPR52688.2022.01553 | 2021 | Masked Autoencoders Are Scalable Vision Learners | Kai-Ming He | 6 | Computer Vision and Pattern Recognition | 13366 | This paper shows that masked autoencoders (MAE) are scalable self-supervised learners for computer vision. Our MAE approach is simple: we mask random patches of the input image and reconstruct the missing pixels. It is based on two core designs. First, we develop an asymmetric encoder-decoder architecture, with an encoder that operates only on the visible subset of patches (without mask tokens), along with a lightweight decoder that reconstructs the original image from the latent representation and mask tokens. Second, we find that masking a high proportion of the input image, e.g., 75%, yields a nontrivial and meaningful self-supervisory task. Coupling these two designs enables us to train large models efficiently and effectively: we accelerate training (by 3× or more) and improve accuracy. Our scalable approach allows for learning high-capacity models that generalize well: e.g., a vanilla ViT-Huge model achieves the best accuracy (87.8%) among methods that use only ImageNet-1K data. Transfer performance in downstream tasks outperforms supervised pretraining and shows promising scaling behavior. |
| 11 | 2582a04918f6fe62dc142f2fca9ca0bb0b1d7895 | 2110.09456 | 2021 | NormFormer: Improved Transformer Pretraining with Extra Normalization | Sam Shleifer | 3 | arXiv.org | 102 | During pretraining, the Pre-LayerNorm transformer suffers from a gradient magnitude mismatch: gradients at early layers are much larger than at later layers. These issues can be alleviated by our proposed NormFormer architecture, which adds three normalization operations to each layer: a Layer Norm after self attention, head-wise scaling of self-attention outputs, and a Layer Norm after the first fully connected layer. The extra operations incur negligible compute cost (+0.4% parameter increase), but improve pretraining perplexity and downstream task performance for both causal and masked language models ranging from 125 Million to 2.7 Billion parameters. For example, adding NormFormer on top of our strongest 1.3B parameter baseline can reach equal perplexity 24% faster, or converge 0.27 perplexity better in the same compute budget. This model reaches GPT3-Large (1.3B) zero shot performance 60% faster. For masked language modeling, NormFormer improves fine-tuned GLUE performance by 1.9% on average. Code to train NormFormer models is available in fairseq https://github.com/pytorch/fairseq/tree/main/examples/normformer . | |
| 12 | 83474f6510e0b985595f936d233321e131966bae | 2110.04369 | 2021 | A Loss Curvature Perspective on Training Instability in Deep Learning | J. Gilmer | 9 | arXiv.org | 53 | In this work, we study the evolution of the loss Hessian across many classification tasks in order to understand the effect the curvature of the loss has on the training dynamics. Whereas prior work has focused on how different learning rates affect the loss Hessian observed during training, we also analyze the effects of model initialization, architectural choices, and common training heuristics such as gradient clipping and learning rate warmup. Our results demonstrate that successful model and hyperparameter choices allow the early optimization trajectory to either avoid -- or navigate out of -- regions of high curvature and into flatter regions that tolerate a higher learning rate. Our results suggest a unifying perspective on how disparate mitigation strategies for training instability ultimately address the same underlying failure mode of neural network optimization, namely poor conditioning. Inspired by the conditioning perspective, we show that learning rate warmup can improve training stability just as much as batch normalization, layer normalization, MetaInit, GradInit, and Fixup initialization. | |
| 13 | 2a805d0e1b067444a554c5169d189fa1f649f411 | 2106.04560 | 10.1109/CVPR52688.2022.01179 | 2021 | Scaling Vision Transformers | Xiao-Hua Zhai | 4 | Computer Vision and Pattern Recognition | 1538 | Attention-based neural networks such as the Vision Transformer (ViT) have recently attained state-of-the-art results on many computer vision benchmarks. Scale is a primary ingredient in attaining excellent results, therefore, understanding a model's scaling properties is a key to designing future generations effectively. While the laws for scaling Transformer language models have been studied, it is unknown how Vision Transformers scale. To address this, we scale ViT models and data, both up and down, and characterize the relationships between error rate, data, and compute. Along the way, we refine the architecture and training of ViT, reducing memory consumption and increasing accuracy of the resulting models. As a result, we successfully train a ViT model with two billion parameters, which attains a new state-of-the-art on ImageNet of 90.45% top-1 accuracy. The model also performs well for few-shot transfer, for example, reaching 84.86% top-1 accuracy on ImageNet with only 10 examples per class. |
| 14 | 7509c66a666e2e3f14bc8676b969b945ee6e136f | 2106.03143 | 2021 | CAPE: Encoding Relative Positions with Continuous Augmented Positional Embeddings | Tatiana Likhomanenko | 5 | Neural Information Processing Systems | 74 | Without positional information, attention-based Transformer neural networks are permutation-invariant. Absolute or relative positional embeddings are the most popular ways to feed Transformer models with positional information. Absolute positional embeddings are simple to implement, but suffer from generalization issues when evaluating on sequences longer than seen at training time. Relative positions are more robust to input length change, but are more complex to implement and yield inferior model throughput due to extra computational and memory costs. In this paper, we propose an augmentation-based approach (CAPE) for absolute positional embeddings, which keeps the advantages of both absolute (simplicity and speed) and relative positional embeddings (better generalization). In addition, our empirical evaluation on state-of-the-art models in machine translation, image and speech recognition demonstrates that CAPE leads to better generalization performance as well as increased stability with respect to training hyper-parameters. | |
| 15 | 42a7015e48a1e00b70ebb442a82afb4b10017c0b | 2106.01548 | 2021 | When Vision Transformers Outperform ResNets without Pretraining or Strong Data Augmentations | Xiangning Chen | 3 | International Conference on Learning Representations | 406 | Vision Transformers (ViTs) and MLPs signal further efforts on replacing hand-wired features or inductive biases with general-purpose neural architectures. Existing works empower the models by massive data, such as large-scale pre-training and/or repeated strong data augmentations, and still report optimization-related problems (e.g., sensitivity to initialization and learning rates). Hence, this paper investigates ViTs and MLP-Mixers from the lens of loss geometry, intending to improve the models' data efficiency at training and generalization at inference. Visualization and Hessian reveal extremely sharp local minima of converged models. By promoting smoothness with a recently proposed sharpness-aware optimizer, we substantially improve the accuracy and robustness of ViTs and MLP-Mixers on various tasks spanning supervised, adversarial, contrastive, and transfer learning (e.g., +5.3\% and +11.0\% top-1 accuracy on ImageNet for ViT-B/16 and Mixer-B/16, respectively, with the simple Inception-style preprocessing). We show that the improved smoothness attributes to sparser active neurons in the first few layers. The resultant ViTs outperform ResNets of similar size and throughput when trained from scratch on ImageNet without large-scale pre-training or strong data augmentations. Model checkpoints are available at \url{https://github.com/google-research/vision_transformer}. | |
| 16 | ad4a0938c48e61b7827869e4ac3baffd0aefab35 | 2104.14294 | 10.1109/ICCV48922.2021.00951 | 2021 | Emerging Properties in Self-Supervised Vision Transformers | Mathilde Caron | 7 | IEEE International Conference on Computer Vision | 10753 | In this paper, we question if self-supervised learning provides new properties to Vision Transformer (ViT) [16] that stand out compared to convolutional networks (convnets). Beyond the fact that adapting self-supervised methods to this architecture works particularly well, we make the following observations: first, self-supervised ViT features contain explicit information about the semantic segmentation of an image, which does not emerge as clearly with supervised ViTs, nor with convnets. Second, these features are also excellent k-NN classifiers, reaching 78.3% top-1 on ImageNet with a small ViT. Our study also underlines the importance of momentum encoder [26], multi-crop training [9], and the use of small patches with ViTs. We implement our findings into a simple self-supervised method, called DINO, which we interpret as a form of self-distillation with no labels. We show the synergy between DINO and ViTs by achieving 80.1% top-1 on ImageNet in linear evaluation with ViT-Base. |
| 17 | 0d5406775fab3e71848908327fb5504df5f60f92 | 2104.10972 | 2021 | ImageNet-21K Pretraining for the Masses | T. Ridnik | 4 | NeurIPS Datasets and Benchmarks | 1011 | ImageNet-1K serves as the primary dataset for pretraining deep learning models for computer vision tasks. ImageNet-21K dataset, which is bigger and more diverse, is used less frequently for pretraining, mainly due to its complexity, low accessibility, and underestimation of its added value. This paper aims to close this gap, and make high-quality efficient pretraining on ImageNet-21K available for everyone. Via a dedicated preprocessing stage, utilization of WordNet hierarchical structure, and a novel training scheme called semantic softmax, we show that various models significantly benefit from ImageNet-21K pretraining on numerous datasets and tasks, including small mobile-oriented models. We also show that we outperform previous ImageNet-21K pretraining schemes for prominent new models like ViT and Mixer. Our proposed pretraining pipeline is efficient, accessible, and leads to SoTA reproducible results, from a publicly available dataset. The training code and pretrained models are available at: https://github.com/Alibaba-MIIL/ImageNet21K | |
| 18 | 739ceacfafb1c4eaa17509351b647c773270b3ae | 2104.02057 | 10.1109/ICCV48922.2021.00950 | 2021 | An Empirical Study of Training Self-Supervised Vision Transformers | Xinlei Chen | 3 | IEEE International Conference on Computer Vision | 2549 | This paper does not describe a novel method. Instead, it studies a straightforward, incremental, yet must-know baseline given the recent progress in computer vision: self-supervised learning for Vision Transformers (ViT). While the training recipes for standard convolutional networks have been highly mature and robust, the recipes for ViT are yet to be built, especially in the self-supervised scenarios where training becomes more challenging. In this work, we go back to basics and investigate the effects of several fundamental components for training self-supervised ViT. We observe that instability is a major issue that degrades accuracy, and it can be hidden by apparently good results. We reveal that these results are indeed partial failure, and they can be improved when training is made more stable. We benchmark ViT results in MoCo v3 and several other self-supervised frameworks, with ablations in various aspects. We discuss the currently positive evidence as well as challenges and open questions. We hope that this work will provide useful data points and experience for future research. |
| 19 | dfb37e6216e792bf6bd5a30c0fc7ad55df1cb71e | 2103.03404 | 2021 | Attention is Not All You Need: Pure Attention Loses Rank Doubly Exponentially with Depth | Yihe Dong | 3 | International Conference on Machine Learning | 643 | Attention-based architectures have become ubiquitous in machine learning, yet our understanding of the reasons for their effectiveness remains limited. This work proposes a new way to understand self-attention networks: we show that their output can be decomposed into a sum of smaller terms, each involving the operation of a sequence of attention heads across layers. Using this decomposition, we prove that self-attention possesses a strong inductive bias towards"token uniformity". Specifically, without skip connections or multi-layer perceptrons (MLPs), the output converges doubly exponentially to a rank-1 matrix. On the other hand, skip connections and MLPs stop the output from degeneration. Our experiments verify the identified convergence phenomena on different variants of standard transformer architectures. | |
| 20 | 1480446723a1aefc26ebb0e6f105c35ab52ddb42 | 2103.00065 | 2021 | Gradient Descent on Neural Networks Typically Occurs at the Edge of Stability | Jeremy M. Cohen | 5 | International Conference on Learning Representations | 522 | We empirically demonstrate that full-batch gradient descent on neural network training objectives typically operates in a regime we call the Edge of Stability. In this regime, the maximum eigenvalue of the training loss Hessian hovers just above the numerical value $2 / \text{(step size)}$, and the training loss behaves non-monotonically over short timescales, yet consistently decreases over long timescales. Since this behavior is inconsistent with several widespread presumptions in the field of optimization, our findings raise questions as to whether these presumptions are relevant to neural network training. We hope that our findings will inspire future efforts aimed at rigorously understanding optimization at the Edge of Stability. Code is available at https://github.com/locuslab/edge-of-stability. | |
| 21 | 2b8088253e2378fce001a090fe923b81e8dedf25 | 2101.03697 | 10.1109/CVPR46437.2021.01352 | 2021 | RepVGG: Making VGG-style ConvNets Great Again | Xiaohan Ding | 6 | Computer Vision and Pattern Recognition | 2619 | We present a simple but powerful architecture of convolutional neural network, which has a VGG-like inference-time body composed of nothing but a stack of 3 × 3 convolution and ReLU, while the training-time model has a multi-branch topology. Such decoupling of the training-time and inference-time architecture is realized by a structural re-parameterization technique so that the model is named RepVGG. On ImageNet, RepVGG reaches over 80% top-1 accuracy, which is the first time for a plain model, to the best of our knowledge. On NVIDIA 1080Ti GPU, RepVGG models run 83% faster than ResNet-50 or 101% faster than ResNet-101 with higher accuracy and show favorable accuracy-speed trade-off compared to the state-of-the-art models like EfficientNet and RegNet. The code and trained models are available at https://github.com/megvii-model/RepVGG. |
| 22 | 0822f8d7e6a72a65e65f147d3a8d8fccd485da40 | 2012.15832 | 10.18653/v1/2021.acl-long.427 | 2021 | Shortformer: Better Language Modeling using Shorter Inputs | Ofir Press | 3 | Annual Meeting of the Association for Computational Linguistics | 104 | Increasing the input length has been a driver of progress in language modeling with transformers. We identify conditions where shorter inputs are not harmful, and achieve perplexity and efficiency improvements through two new methods that decrease input length. First, we show that initially training a model on short subsequences before moving on to longer ones both reduces overall training time and, surprisingly, substantially improves perplexity. Second, we show how to improve the efficiency of recurrence methods in transformers, which let models condition on previously processed tokens when generating sequences that exceed the maximal length the transformer can handle at once. Existing methods require computationally expensive relative position embeddings; we introduce a simple alternative of adding absolute position embeddings to queries and keys instead of to word embeddings, which efficiently produces superior results. We show that these recurrent models also benefit from short input lengths. Combining these techniques speeds up training by a factor of 1.65, reduces memory usage, and substantially improves perplexity on WikiText-103, without adding any parameters. |
| 23 | ad7ddcc14984caae308c397f1a589aae75d4ab71 | 2012.12877 | 2020 | Training data-efficient image transformers & distillation through attention | Hugo Touvron | 6 | International Conference on Machine Learning | 10113 | Recently, neural networks purely based on attention were shown to address image understanding tasks such as image classification. However, these visual transformers are pre-trained with hundreds of millions of images using an expensive infrastructure, thereby limiting their adoption. In this work, we produce a competitive convolution-free transformer by training on Imagenet only. We train them on a single computer in less than 3 days. Our reference vision transformer (86M parameters) achieves top-1 accuracy of 83.1% (single-crop evaluation) on ImageNet with no external data. More importantly, we introduce a teacher-student strategy specific to transformers. It relies on a distillation token ensuring that the student learns from the teacher through attention. We show the interest of this token-based distillation, especially when using a convnet as a teacher. This leads us to report results competitive with convnets for both Imagenet (where we obtain up to 85.2% accuracy) and when transferring to other tasks. We share our code and models. | |
| 24 | 62ce4d65335c32844247e0ecf3be6de1ccb924b2 | 2010.11745 | 10.21437/interspeech.2021-1758 | 2020 | Rethinking Evaluation in ASR: Are Our Models Robust Enough? | Tatiana Likhomanenko | 8 | Interspeech | 117 | Is pushing numbers on a single benchmark valuable in automatic speech recognition? Research results in acoustic modeling are typically evaluated based on performance on a single dataset. While the research community has coalesced around various benchmarks, we set out to understand generalization performance in acoustic modeling across datasets -- in particular, if models trained on a single dataset transfer to other (possibly out-of-domain) datasets. Further, we demonstrate that when a large enough set of benchmarks is used, average word error rate (WER) performance over them provides a good proxy for performance on real-world data. Finally, we show that training a single acoustic model on the most widely-used datasets -- combined -- reaches competitive performance on both research and real-world benchmarks. |
| 25 | 7f0c7c324675179f0e32c160d99c7066c7ab30ae | 2010.11524 | 10.21437/interspeech.2021-740 | 2020 | slimIPL: Language-Model-Free Iterative Pseudo-Labeling | Tatiana Likhomanenko | 5 | Interspeech | 79 | Recent results in end-to-end ASR have demonstrated the efficacy of simple pseudo-labeling for semi-supervised models trained both with Connectionist Temporal Classification (CTC) and Sequence-to-Sequence (seq2seq) losses. Iterative Pseudo-Labeling (IPL), which continuously trains a single model using pseudo-labels iteratively re-generated as the model learns, has been shown to further increase performance in ASR. We improve upon the IPL algorithm: as the model learns, we propose to iteratively re-generate transcriptions with hard labels (the most probable tokens) assignments, that is without a language model. We call this approach Language-Model-Free IPL (slimIPL) and we give a resultant training setup for CTC and seq2seq models. At inference, our experiments show that decoding with a strong language model is more beneficial with slimIPL than IPL, asIPL exhibits some language model over-fitting issues. Compared to prior work on semi-supervised and unsupervised approaches, slimIPL not only simplifies the training process, but also achieves competitive and state-of-the-art results on LibriSpeech test sets in both standard and low-resource settings. |
| 26 | 268d347e8a55b5eb82fb5e7d2f800e33c75ab18a | 2010.11929 | 2020 | An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale | A. Dosovitskiy | 12 | International Conference on Learning Representations | 72694 | While the Transformer architecture has become the de-facto standard for natural language processing tasks, its applications to computer vision remain limited. In vision, attention is either applied in conjunction with convolutional networks, or used to replace certain components of convolutional networks while keeping their overall structure in place. We show that this reliance on CNNs is not necessary and a pure transformer applied directly to sequences of image patches can perform very well on image classification tasks. When pre-trained on large amounts of data and transferred to multiple mid-sized or small image recognition benchmarks (ImageNet, CIFAR-100, VTAB, etc.), Vision Transformer (ViT) attains excellent results compared to state-of-the-art convolutional networks while requiring substantially fewer computational resources to train. | |
| 27 | a2cd073b57be744533152202989228cb4122270a | 2010.01412 | 2020 | Sharpness-Aware Minimization for Efficiently Improving Generalization | Pierre Foret | 4 | International Conference on Learning Representations | 2174 | In today's heavily overparameterized models, the value of the training loss provides few guarantees on model generalization ability. Indeed, optimizing only the training loss value, as is commonly done, can easily lead to suboptimal model quality. Motivated by the connection between geometry of the loss landscape and generalization---including a generalization bound that we prove here---we introduce a novel, effective procedure for instead simultaneously minimizing loss value and loss sharpness. In particular, our procedure, Sharpness-Aware Minimization (SAM), seeks parameters that lie in neighborhoods having uniformly low loss; this formulation results in a min-max optimization problem on which gradient descent can be performed efficiently. We present empirical results showing that SAM improves model generalization across a variety of benchmark datasets (e.g., CIFAR-{10, 100}, ImageNet, finetuning tasks) and models, yielding novel state-of-the-art performance for several. Additionally, we find that SAM natively provides robustness to label noise on par with that provided by state-of-the-art procedures that specifically target learning with noisy labels. | |
| 28 | b5271f4522fd72e335535c5f65d3afc01d1cb2bd | 2008.07772 | 2020 | Very Deep Transformers for Neural Machine Translation | Xiao-Dong Liu | 4 | arXiv.org | 111 | We explore the application of very deep Transformer models for Neural Machine Translation (NMT). Using a simple yet effective initialization technique that stabilizes training, we show that it is feasible to build standard Transformer-based models with up to 60 encoder layers and 12 decoder layers. These deep models outperform their baseline 6-layer counterparts by as much as 2.5 BLEU, and achieve new state-of-the-art benchmark results on WMT14 English-French (43.8 BLEU and 46.4 BLEU with back-translation) and WMT14 English-German (30.1 BLEU).The code and trained models will be publicly available at: this https URL. | |
| 29 | e00484961fb2f30d2d48a5f9853fa3ebab140cac | 2020 | Improving Transformer Optimization Through Better Initialization | Xiao-Shi Huang | 4 | International Conference on Machine Learning | 194 | |||
| 30 | 8d908042f139575d6688c745e94156c9df6eae07 | 2004.08249 | 10.18653/v1/2020.emnlp-main.463 | 2020 | Understanding the Difficulty of Training Transformers | Liyuan Liu | 5 | Conference on Empirical Methods in Natural Language Processing | 328 | Transformers have been proved effective for many deep learning tasks. Training transformers, however, requires non-trivial efforts regarding carefully designing learning rate schedulers and cutting-edge optimizers (the standard SGD fails to train Transformers effectively). In this paper, we study Transformer training from both theoretical and empirical perspectives. Our analysis reveals that unbalanced gradients are not the root cause of the instability of training. Instead, we identify an amplification effect that substantially influences training. Specifically, we observe that for each layer in a multi-layer Transformer model, heavy dependency on its residual branch makes training unstable since it amplifies small parameter perturbations (e.g., parameter updates) and result in significant disturbances in the model output, yet a light dependency limits the potential of model training and can lead to an inferior trained model. Inspired by our analysis, we propose Admin ($\mathbf{Ad}$aptive $\mathbf{m}$odel $\mathbf{in}$itialization) to stabilize the training in the early stage and unleash its full potential in the late stage. Extensive experiments show that Admin is more stable, converges faster, and leads to better performance. |
| 31 | e52051204cb1179584f3b008c9d38848b52c1f28 | 2003.04887 | 2020 | ReZero is All You Need: Fast Convergence at Large Depth | Thomas C. Bachlechner | 5 | Conference on Uncertainty in Artificial Intelligence | 462 | Deep networks often suffer from vanishing or exploding gradients due to inefficient signal propagation, leading to long training times or convergence difficulties. Various architecture designs, sophisticated residual-style networks, and initialization schemes have been shown to improve deep signal propagation. Recently, Pennington et al. used free probability theory to show that dynamical isometry plays an integral role in efficient deep learning. We show that the simplest architecture change of gating each residual connection using a single zero-initialized parameter satisfies initial dynamical isometry and outperforms more complex approaches. Although much simpler than its predecessors, this gate enables training thousands of fully connected layers with fast convergence and better test performance for ResNets trained on CIFAR-10. We apply this technique to language modeling and find that we can easily train 120-layer Transformers. When applied to 12 layer Transformers, it converges 56% faster on enwiki8. | |
| 32 | 7af72a461ed7cda180e7eab878efd5f35d79bbf4 | 2002.05709 | 2020 | A Simple Framework for Contrastive Learning of Visual Representations | Ting Chen | 4 | International Conference on Machine Learning | 26915 | This paper presents SimCLR: a simple framework for contrastive learning of visual representations. We simplify recently proposed contrastive self-supervised learning algorithms without requiring specialized architectures or a memory bank. In order to understand what enables the contrastive prediction tasks to learn useful representations, we systematically study the major components of our framework. We show that (1) composition of data augmentations plays a critical role in defining effective predictive tasks, (2) introducing a learnable nonlinear transformation between the representation and the contrastive loss substantially improves the quality of the learned representations, and (3) contrastive learning benefits from larger batch sizes and more training steps compared to supervised learning. By combining these findings, we are able to considerably outperform previous methods for self-supervised and semi-supervised learning on ImageNet. A linear classifier trained on self-supervised representations learned by SimCLR achieves 76.5% top-1 accuracy, which is a 7% relative improvement over previous state-of-the-art, matching the performance of a supervised ResNet-50. When fine-tuned on only 1% of the labels, we achieve 85.8% top-5 accuracy, outperforming AlexNet with 100X fewer labels. | |
| 33 | 6e5d89c2b3b5ead2c3ab389534de62a28c1e8e6e | 1912.07145 | 10.1109/BigData50022.2020.9378171 | 2019 | PyHessian: Neural Networks Through the Lens of the Hessian | Z. Yao | 4 | 2020 IEEE International Conference on Big Data (Big Data) | 423 | We present PYHESSIAN, a new scalable framework that enables fast computation of Hessian (i.e., second-order derivative) information for deep neural networks. PYHESSIAN enables fast computations of the top Hessian eigenvalues, the Hessian trace, and the full Hessian eigenvalue/spectral density; it supports distributed-memory execution on cloud/supercomputer systems; and it is available as open source [1]. This general framework can be used to analyze neural network models, including the topology of the loss landscape (i.e., curvature information) to gain insight into the behavior of different models/optimizers. As an example, we analyze the effect of residual connections and Batch Normalization layers on the trainability of neural networks. One recent claim, based on simpler first-order analysis, is that residual connections and Batch Normalization make the loss landscape "smoother," thus making it easier for Stochastic Gradient Descent to converge to a good solution. Our second-order analysis, easily enabled by PYHESSIAN, shows new finer-scale insights, demonstrating that while conventional wisdom is sometimes validated, in other cases it is simply incorrect. In particular, we find that Batch Normalization does not necessarily make the loss landscape smoother, especially for shallow networks. |
| 34 | 703685e969fed715e13937c11d7ecc5cc7c4dfd0 | 1910.05895 | 10.5281/zenodo.3525484 | 2019 | Transformers without Tears: Improving the Normalization of Self-Attention | Toan Q. Nguyen | 2 | International Workshop on Spoken Language Translation | 288 | We evaluate three simple, normalization-centric changes to improve Transformer training. First, we show that pre-norm residual connections (PRENORM) and smaller initializations enable warmup-free, validation-based training with large learning rates. Second, we propose l2 normalization with a single scale parameter (SCALENORM) for faster training and better performance. Finally, we reaffirm the effectiveness of normalizing word embeddings to a fixed length (FIXNORM). On five low-resource translation pairs from TED Talks-based corpora, these changes always converge, giving an average +1.1 BLEU over state-of-the-art bilingual baselines and a new 32.8 BLEU on IWSLT '15 English-Vietnamese. We ob- serve sharper performance curves, more consistent gradient norms, and a linear relationship between activation scaling and decoder depth. Surprisingly, in the high-resource setting (WMT '14 English-German), SCALENORM and FIXNORM remain competitive but PRENORM degrades performance. |
| 35 | b0fae9fbb4e580d92395eabafe73e317ae6510e3 | 1904.08779 | 10.21437/Interspeech.2019-2680 | 2019 | SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognition | Daniel S. Park | 7 | Interspeech | 4310 | We present SpecAugment, a simple data augmentation method for speech recognition. SpecAugment is applied directly to the feature inputs of a neural network (i.e., filter bank coefficients). The augmentation policy consists of warping the features, masking blocks of frequency channels, and masking blocks of time steps. We apply SpecAugment on Listen, Attend and Spell networks for end-to-end speech recognition tasks. We achieve state-of-the-art performance on the LibriSpeech 960h and Swichboard 300h tasks, outperforming all prior work. On LibriSpeech, we achieve 6.8% WER on test-other without the use of a language model, and 5.8% WER with shallow fusion with a language model. This compares to the previous state-of-the-art hybrid system of 7.5% WER. For Switchboard, we achieve 7.2%/14.6% on the Switchboard/CallHome portion of the Hub5'00 test set without the use of a language model, and 6.8%/14.1% with shallow fusion, which compares to the previous state-of-the-art hybrid system at 8.3%/17.3% WER. |
| 36 | faadd7d081c8d67e8c2567e8a5579e46cd6b2280 | 1904.01038 | 10.18653/v1/N19-4009 | 2019 | fairseq: A Fast, Extensible Toolkit for Sequence Modeling | Myle Ott | 8 | North American Chapter of the Association for Computational Linguistics | 3444 | fairseq is an open-source sequence modeling toolkit that allows researchers and developers to train custom models for translation, summarization, language modeling, and other text generation tasks. The toolkit is based on PyTorch and supports distributed training across multiple GPUs and machines. We also support fast mixed-precision training and inference on modern GPUs. A demo video can be found at https://www.youtube.com/watch?v=OtgDdWtHvto |
| 37 | 409b9abbeecbc91d00187e73d441f556c8c6b598 | 1901.10159 | 2019 | An Investigation into Neural Net Optimization via Hessian Eigenvalue Density | B. Ghorbani | 3 | International Conference on Machine Learning | 494 | To understand the dynamics of optimization in deep neural networks, we develop a tool to study the evolution of the entire Hessian spectrum throughout the optimization process. Using this, we study a number of hypotheses concerning smoothness, curvature, and sharpness in the deep learning literature. We then thoroughly analyze a crucial structural feature of the spectra: in non-batch normalized networks, we observe the rapid appearance of large isolated eigenvalues in the spectrum, along with a surprising concentration of the gradient in the corresponding eigenspaces. In batch normalized networks, these two effects are almost absent. We characterize these effects, and explain how they affect optimization speed through both theory and experiments. As part of this work, we adapt advanced tools from numerical linear algebra that allow scalable and accurate estimation of the entire Hessian spectrum of ImageNet-scale neural networks; this technique may be of independent interest in other applications. | |
| 38 | 96c82727dd5a80fef93007f888bb8569feb6bd85 | 1901.09321 | 2019 | Fixup Initialization: Residual Learning Without Normalization | Hongyi Zhang | 3 | International Conference on Learning Representations | 430 | Normalization layers are a staple in state-of-the-art deep neural network architectures. They are widely believed to stabilize training, enable higher learning rate, accelerate convergence and improve generalization, though the reason for their effectiveness is still an active research topic. In this work, we challenge the commonly-held beliefs by showing that none of the perceived benefits is unique to normalization. Specifically, we propose fixed-update initialization (Fixup), an initialization motivated by solving the exploding and vanishing gradient problem at the beginning of training via properly rescaling a standard initialization. We find training residual networks with Fixup to be as stable as training with normalization -- even for networks with 10,000 layers. Furthermore, with proper regularization, Fixup enables residual networks without normalization to achieve state-of-the-art performance in image classification and machine translation. | |
| 39 | d170bd486e4c0fe82601e322b0e9e0dde63ab299 | 1809.10853 | 2018 | Adaptive Input Representations for Neural Language Modeling | Alexei Baevski | 2 | International Conference on Learning Representations | 450 | We introduce adaptive input representations for neural language modeling which extend the adaptive softmax of Grave et al. (2017) to input representations of variable capacity. There are several choices on how to factorize the input and output layers, and whether to model words, characters or sub-word units. We perform a systematic comparison of popular choices for a self-attentional architecture. Our experiments show that models equipped with adaptive embeddings are more than twice as fast to train than the popular character input CNN while having a lower number of parameters. On the WikiText-103 benchmark we achieve 18.7 perplexity, an improvement of 10.5 perplexity compared to the previously best published result and on the Billion Word benchmark, we achieve 23.02 perplexity. | |
| 40 | c8efcc854d97dfc2a42b83316a2109f9d166e43f | 1803.02155 | 10.18653/v1/N18-2074 | 2018 | Self-Attention with Relative Position Representations | Peter Shaw | 3 | North American Chapter of the Association for Computational Linguistics | 3001 | Relying entirely on an attention mechanism, the Transformer introduced by Vaswani et al. (2017) achieves state-of-the-art results for machine translation. In contrast to recurrent and convolutional neural networks, it does not explicitly model relative or absolute position information in its structure. Instead, it requires adding representations of absolute positions to its inputs. In this work we present an alternative approach, extending the self-attention mechanism to efficiently consider representations of the relative positions, or distances between sequence elements. On the WMT 2014 English-to-German and English-to-French translation tasks, this approach yields improvements of 1.3 BLEU and 0.3 BLEU over absolute position representations, respectively. Notably, we observe that combining relative and absolute position representations yields no further improvement in translation quality. We describe an efficient implementation of our method and cast it as an instance of relation-aware self-attention mechanisms that can generalize to arbitrary graph-labeled inputs. |
| 41 | 84de7d27e2f6160f634a483e8548c499a2cda7fa | 1802.05957 | 2018 | Spectral Normalization for Generative Adversarial Networks | Takeru Miyato | 4 | International Conference on Learning Representations | 5267 | One of the challenges in the study of generative adversarial networks is the instability of its training. In this paper, we propose a novel weight normalization technique called spectral normalization to stabilize the training of the discriminator. Our new normalization technique is computationally light and easy to incorporate into existing implementations. We tested the efficacy of spectral normalization on CIFAR10, STL-10, and ILSVRC2012 dataset, and we experimentally confirmed that spectrally normalized GANs (SN-GANs) is capable of generating images of better or equal quality relative to the previous training stabilization techniques. | |
| 42 | 1e3d18beaf3921f561e1b999780f29f2b23f3b7d | 1708.03888 | 2017 | Large Batch Training of Convolutional Networks | Yang You | 3 | 995 | A common way to speed up training of large convolutional networks is to add computational units. Training is then performed using data-parallel synchronous Stochastic Gradient Descent (SGD) with mini-batch divided between computational units. With an increase in the number of nodes, the batch size grows. But training with large batch size often results in the lower model accuracy. We argue that the current recipe for large batch training (linear learning rate scaling with warm-up) is not general enough and training may diverge. To overcome this optimization difficulties we propose a new training algorithm based on Layer-wise Adaptive Rate Scaling (LARS). Using LARS, we scaled Alexnet up to a batch size of 8K, and Resnet-50 to a batch size of 32K without loss in accuracy. | ||
| 43 | 204e3073870fae3d05bcbc2f6a8e263d9b72e776 | 1706.03762 | 2017 | Attention is All you Need | Ashish Vaswani | 8 | Neural Information Processing Systems | 199000 | The dominant sequence transduction models are based on complex recurrent or convolutional neural networks in an encoder-decoder configuration. The best performing models also connect the encoder and decoder through an attention mechanism. We propose a new simple network architecture, the Transformer, based solely on attention mechanisms, dispensing with recurrence and convolutions entirely. Experiments on two machine translation tasks show these models to be superior in quality while being more parallelizable and requiring significantly less time to train. Our model achieves 28.4 BLEU on the WMT 2014 English-to-German translation task, improving over the existing best results, including ensembles by over 2 BLEU. On the WMT 2014 English-to-French translation task, our model establishes a new single-model state-of-the-art BLEU score of 41.8 after training for 3.5 days on eight GPUs, a small fraction of the training costs of the best models from the literature. We show that the Transformer generalizes well to other tasks by applying it successfully to English constituency parsing both with large and limited training data. | |
| 44 | efbd381493bb9636f489b965a2034d529cd56bcd | 1609.07843 | 2016 | Pointer Sentinel Mixture Models | Stephen Merity | 4 | International Conference on Learning Representations | 4957 | Recent neural network sequence models with softmax classifiers have achieved their best language modeling performance only with very large hidden states and large vocabularies. Even then they struggle to predict rare or unseen words even if the context makes the prediction unambiguous. We introduce the pointer sentinel mixture architecture for neural sequence models which has the ability to either reproduce a word from the recent context or produce a word from a standard softmax classifier. Our pointer sentinel-LSTM model achieves state of the art language modeling performance on the Penn Treebank (70.9 perplexity) while using far fewer parameters than a standard softmax LSTM. In order to evaluate how well language models can exploit longer contexts and deal with more realistic vocabularies and larger corpora we also introduce the freely available WikiText corpus. | |
| 45 | 63e39cdf1ad884da6bc69096bb3413b5b1100559 | 1608.05859 | 10.18653/V1/E17-2025 | 2016 | Using the Output Embedding to Improve Language Models | Ofir Press | 2 | Conference of the European Chapter of the Association for Computational Linguistics | 871 | We study the topmost weight matrix of neural network language models. We show that this matrix constitutes a valid word embedding. When training language models, we recommend tying the input embedding and this output embedding. We analyze the resulting update rules and show that the tied embedding evolves in a more similar way to the output embedding than to the input embedding in the untied model. We also offer a new method of regularizing the output embedding. Our methods lead to a significant reduction in perplexity, as we are able to show on a variety of neural network language models. Finally, we show that weight tying can reduce the size of neural translation models to less than half of their original size without harming their performance. |
| 46 | 97fb4e3d45bb098e27e0071448b6152217bd35a5 | 1607.06450 | 2016 | Layer Normalization | Jimmy Ba | 3 | arXiv.org | 13427 | Training state-of-the-art, deep neural networks is computationally expensive. One way to reduce the training time is to normalize the activities of the neurons. A recently introduced technique called batch normalization uses the distribution of the summed input to a neuron over a mini-batch of training cases to compute a mean and variance which are then used to normalize the summed input to that neuron on each training case. This significantly reduces the training time in feed-forward neural networks. However, the effect of batch normalization is dependent on the mini-batch size and it is not obvious how to apply it to recurrent neural networks. In this paper, we transpose batch normalization into layer normalization by computing the mean and variance used for normalization from all of the summed inputs to the neurons in a layer on a single training case. Like batch normalization, we also give each neuron its own adaptive bias and gain which are applied after the normalization but before the non-linearity. Unlike batch normalization, layer normalization performs exactly the same computation at training and test times. It is also straightforward to apply to recurrent neural networks by computing the normalization statistics separately at each time step. Layer normalization is very effective at stabilizing the hidden state dynamics in recurrent networks. Empirically, we show that layer normalization can substantially reduce the training time compared with previously published techniques. | |
| 47 | 3d2c6941a9b4608ba52b328369a3352db2092ae0 | 1602.07868 | 2016 | Weight Normalization: A Simple Reparameterization to Accelerate Training of Deep Neural Networks | T. Salimans | 2 | Neural Information Processing Systems | 2197 | We present weight normalization: a reparameterization of the weight vectors in a neural network that decouples the length of those weight vectors from their direction. By reparameterizing the weights in this way we improve the conditioning of the optimization problem and we speed up convergence of stochastic gradient descent. Our reparameterization is inspired by batch normalization but does not introduce any dependencies between the examples in a minibatch. This means that our method can also be applied successfully to recurrent models such as LSTMs and to noise-sensitive applications such as deep reinforcement learning or generative models, for which batch normalization is less well suited. Although our method is much simpler, it still provides much of the speed-up of full batch normalization. In addition, the computational overhead of our method is lower, permitting more optimization steps to be taken in the same amount of time. We demonstrate the usefulness of our method on applications in supervised image recognition, generative modelling, and deep reinforcement learning. | |
| 48 | 34038d9424ce602d7ac917a4e582d977725d4393 | 10.1109/ICASSP.2015.7178964 | 2015 | Librispeech: An ASR corpus based on public domain audio books | Vassil Panayotov | 4 | IEEE International Conference on Acoustics, Speech, and Signal Processing | 8673 | ||
| 49 | a6cb366736791bcccc5c8639de5a8f9636bf87e8 | 1412.6980 | 2014 | Adam: A Method for Stochastic Optimization | Diederik P. Kingma | 2 | International Conference on Learning Representations | 173229 | We introduce Adam, an algorithm for first-order gradient-based optimization of stochastic objective functions, based on adaptive estimates of lower-order moments. The method is straightforward to implement, is computationally efficient, has little memory requirements, is invariant to diagonal rescaling of the gradients, and is well suited for problems that are large in terms of data and/or parameters. The method is also appropriate for non-stationary objectives and problems with very noisy and/or sparse gradients. The hyper-parameters have intuitive interpretations and typically require little tuning. Some connections to related algorithms, on which Adam was inspired, are discussed. We also analyze the theoretical convergence properties of the algorithm and provide a regret bound on the convergence rate that is comparable to the best known results under the online convex optimization framework. Empirical results demonstrate that Adam works well in practice and compares favorably to other stochastic optimization methods. Finally, we discuss AdaMax, a variant of Adam based on the infinity norm. | |
| 50 | 413c1142de9d91804d6d11c67ff3fed59c9fc279 | 10.5555/1953048.2021068 | 2011 | Adaptive Subgradient Methods for Online Learning and Stochastic Optimization | John C. Duchi | 3 | Journal of machine learning research | 11646 | ||
| 51 | d2c733e34d48784a37d717fe43d9e93277a8c53e | 10.1109/CVPR.2009.5206848 | 2009 | ImageNet: A large-scale hierarchical image database | Jia Deng | 6 | 2009 IEEE Conference on Computer Vision and Pattern Recognition | 77302 | ||
| 52 | a0117b72a4f68d0a134e24f674ca7fd0b42663b7 | 10.1177/001258063805600103 | 1938 | Causality | Illtyd Trethowan | 1 | 2213 | |||
| 53 | 2023 | Results for MT on WMT’17 English-German data for post-LN, with or without additional σReparam, with or without residual rescaling (‘DeepNorm | 0 | A PREPRINT - MARCH | 0 | |||||
| 54 | 2022 | DeepNorm, solution (it uses post-LN and rescale residual connections depending on | 0 | 0 | ||||||
| 55 | 2022 | can recover from attention entropy collapse (happens in encoder layers) and nicely converge, 100L-100L suffers from it and can diverge | 0 | 0 | ||||||
| 56 | 2022 | The goal of this section is to understand how varying the model depth for the well-established recipes affects the training stability | 0 | 0 | ||||||
| 57 | 2022 | We also observed the same trend of decreasing model performance with increasing the model depth. Attention entropy is nicely bounded across all depths similarly to ASR11 | 0 | 0 | ||||||
| 58 | 2021 | Our main stability experiments use 40 epochs of learning rate warmup, matching the setting of Chen et al. | 0 | For more details, see Appendix D.2 | 0 | |||||
| 59 | 2021 | Reduced Learning Rate Warmup | 0 | 0 | ||||||
| 60 | 2021 | 2021) pointed out by Vimal Thilak. All investigations, experiments and related analysis done by Dan Busbridge | 0 | 0 | ||||||
| 61 | 2021 | 2021) the authors noted that the learning rate warmup period needed extending from its typical | 0 | 0 | ||||||
| 62 | 2020 | 50L-50L and 100L-100L, are unable to train and we observe the same vanishing gradients problem as reported by Wang et al | 0 | 0 | ||||||
| 63 | 2020 | 2020a) are recent works that proposed to rescale residual connections | 0 | 0 | ||||||
| 64 | 9405cc0d6169988371b2755e573cc28650d14dfe | 2019 | Language Models are Unsupervised Multitask Learners | A. Radford | 6 | 30929 | ||||
| 65 | 2019 | For our iterative method we use the implementation of Lanczos from CuPy | 0 | Okuta et al | 0 | |||||
| 66 | a59da4639436f582e483347a4833e7659fd3e598 | 2017 | CuPy : A NumPy-Compatible Library for NVIDIA GPU Calculations | Ryosuke Okuta | 5 | 674 | ||||
| 67 | 2e0f0bbe0a630c36ccd42e3b1b62eeb5e8dc4541 | 10.1002/ZAMM.19290090206 | Praktische Verfahren der Gleichungsauflösung . | R. Mises | 2 | 370 | ||||
| 68 | 261a056f8b21918e8616a429b2df6e1d5d33be41 | Connectionist Temporal Classification: Labelling Unsegmented Sequence Data with Recurrent Neural Networks | A. Graves | 4 | 5295 | |||||
| 69 | 845ee9838c1f5bf63b7db2c95ec5d27af14a4e02 | Connectionist Temporal Classification: Labelling Unsegmented Sequences with Recurrent Neural Networks | 0 | 940 | ||||||
| 70 | 2022) results to probe for entropy collapse phenomenon. All later experiments, Section 4.3 and Appendix F, with deep transformers and deepnorm are done by Tatiana Likhomanenko | 0 | 0 | |||||||
| 71 | In vision we initialize the σReparam γ term using the first singular value, computed with the SVD at weight initialization. We then use one power iteration for all further updates | 0 | 0 | |||||||
| 72 | Automatic Speech Recognition All speech recognition experiments are done by Tatiana Likhomanenko. Shuangfei Zhai and Jason Ramapuram advised to have also large scale results | 0 | 0 | |||||||
| 73 | Language Modeling Initial implementation and preliminary results on applicability of σReparam to the language modeling, Appendix G, are done by Yizhe Zhang with help from Jiatao Gu | 0 | 0 |