Sparse Mixture-of-Experts (MoE) architectures route each token through a subset of experts in each layer independently. We propose to look at the calculation of MoE using the lens of expert methods—the sequence of expert choices a token makes across layers. This theory shows that, despite N^L possible ways of N experts in all L layers, the tokens used converge into a small part of the ways relevant to the language task, yet most of the ways remain untested, representing statistical inefficiency. This encourages structures that block the effective area of the path to increase this natural concentration. As one summary, we present PathMoE, which shares router parameters across blocks in successive layers. The analysis confirms that PathMoE maximizes the structure of the emerging pathway: it produces concentrated pathway clusters, better cross-layer consistency, and greater robustness to pathway perturbations. Tests of the 0.9B and 16B PathMoE parameter models show consistent improvements in confusion and downstream functions over independent pathways, while eliminating the need for auxiliary loss. These results establish expert methods as a useful design axis for MoE architectures, complementing existing work on independent route methods.
- ** Work done while at Apple
