Expand description
Binding one GLM MLP site for a rank: TP slicing of the dense/shared halves, EP selection of the routed experts.
Takes load closures rather than a WeightStore, for the same reason
crate::layers::glm5next_dsa::build does: the slicing is then testable without a
checkpoint, and the loader wiring stays one call site.
§🔴 The two axes are different, and mixing them is silent
- TP splits the width of the dense FFN and the shared expert.
gate_proj/up_projare[inter, hidden]and split by ROW;down_projis[hidden, inter]and splits by COLUMN. Slicingdown_projby row instead gives a well-formed[hidden/tp, inter]tensor and a plausible, wrong output. - EP splits the set of routed experts. An expert is never cut — it is owned whole. The router stays replicated so every rank selects the same ids.
Functions§
- build_
dense_ mlp - TP-slice and upload one BF16 SwiGLU MLP — a dense layer, or a routed layer’s shared expert.
- build_
moe - Bind one routed MoE site for this rank: replicated router, TP-sharded shared expert, and
exactly the
local_expertsrouted experts this EP rank owns.