最近我在想 LatentMoE 还能不能再往前走一步。
标准 MoE 只激活少量 experts，但每个 expert 通常仍在完整隐藏维度上计算。LatentMoE 先把 token 压到更低维的潜空间，再交给 expert。这样可以减少参数量、计算量和通信开销。
我的疑问是：所有 experts 共用同一个潜空间，会不会限制它们的分工？
如果所有 token 都先经过同一个投影，那么各个 experts 接收到的信息结构基本一致。expert 之间的差异主要来自后续网络。可不同 experts 可能依赖不同类型的信息，单一潜空间未必总是合适。
因此，我想到使用多个低维投影：
不同 experts 可以绑定不同投影，或者由 router 同时选择 expert 和投影。这样，expert 的差异既来自自身参数，也来自输入表示。
这个想法带来几个直接问题。
第一，多个投影可能学到重复内容。可以加入正交、去相关或其他约束，但这些约束不一定真的有利于任务。
第二，投影数量越多，额外参数和计算越大。多个潜空间能补充信息，也会削弱 LatentMoE 原本的效率优势。
第三，共享程度需要权衡。所有 experts 共用一个投影最省参数；每个 expert 使用独立投影最灵活，但成本很高。更现实的设计可能是让一组 experts 共享一个投影。
我现在最关心的问题是：多个低维空间能否学到互补信息，同时保持足够低的开销。
如果答案是肯定的，那么 LatentMoE 的设计重点就不只在压缩维度，还包括潜空间之间的分工方式。这也是我觉得这个方向值得继续研究的原因。
Lately I’ve been wondering whether LatentMoE can take another step forward.
A standard MoE activates only a few experts, but each expert usually still computes over the full hidden dimension. LatentMoE first compresses the token into a lower-dimensional latent space and then hands it to the expert. This reduces the parameter count, computation, and communication overhead.
My question is this: if all experts share the same latent space, does that limit their division of labor?
If every token first passes through the same projection, then the information each expert receives has essentially the same structure. The differences between experts come mainly from the networks that follow. But different experts may depend on different kinds of information, and a single latent space is not always a good fit.
So I started thinking about using multiple low-dimensional projections:
Different experts can be bound to different projections, or the router can select both the expert and the projection at once. This way, an expert’s distinctiveness comes not only from its own parameters but also from its input representation.
This idea brings a few immediate problems.
First, multiple projections may learn redundant content. We could add orthogonality, decorrelation, or other constraints, but such constraints do not necessarily benefit the task.
Second, the more projections there are, the larger the extra parameters and computation. Multiple latent spaces can add information, but they also weaken the efficiency advantage LatentMoE had in the first place.
Third, the degree of sharing needs to be traded off. Having all experts share one projection is the most parameter-efficient; giving each expert its own projection is the most flexible but very costly. A more realistic design might let a group of experts share one projection.
The question I care about most right now is whether multiple low-dimensional spaces can learn complementary information while keeping the overhead low enough.
If the answer is yes, then the focus of LatentMoE’s design is not only about compressing the dimension, but also about how the latent spaces divide the work. That is why I think this direction is worth pursuing further.