No Parameters Left Behind: Sensitivity Guided Adaptive Learning Rate for Training Large Transformer Models

Chen Liang, Haoming Jiang, Simiao Zuo, Pengcheng He, Xiaodong Liu, Jianfeng Gao, Weizhu Chen, Tuo Zhao
February 2022
Arxiv Github
Type
Conference paper
Publication
The 10th International Conference on Learning Representations (ICLR), 2022