SlimMoE: Structured Compression of Large MoE Models via Expert Slimming and Distillation

Zichong Li, Chen Liang, Zixuan Zhang, Ilgee Hong, Young Jin Kim, Weizhu Chen, Tuo Zhao
July 2025
Arxiv Phi-tiny-MoE-instruct Phi-mini-MoE-instruct Phi-3 Technical Report
Type
Conference paper
Publication
The Second Conference on Language Modeling (COLM), 2025