Workshop · Published ·
StructMoE: Structured Mixture of Experts Using Low Rank Experts
Zain Sarwar, Ashwinee Panda, Benjamin Thérien, Stephen Rawls, Anirban Das, Kartik Balasubramaniam, Berkcan Kapusuzoglu, Shixiong Zhang, Sambit Sahu, Milind Naphade, Supriyo Chakraborty
NeurIPS Efficient Natural Language and Speech Processing Workshop · Workshop
Summary
This paper introduces StructMoE, which augments mixture-of-experts models with dynamic capacity using structured low-rank experts.
Research contribution
StructMoE routes tokens to low-rank expert modules through expert-specific secondary routers and reports held-out loss comparisons with an MoE baseline.