pub struct Glm5NextDsaWeights {Show 13 fields
pub q_a_proj: DevicePtr,
pub q_a_layernorm: DevicePtr,
pub q_absorb: DevicePtr,
pub kv_a_proj: DevicePtr,
pub kv_a_layernorm: DevicePtr,
pub o_absorb: DevicePtr,
pub wk: DevicePtr,
pub k_norm_weight: DevicePtr,
pub k_norm_bias: DevicePtr,
pub compress_gate: DevicePtr,
pub wq_b: DevicePtr,
pub weights_proj: DevicePtr,
pub ape: DevicePtr,
}Expand description
One DSA block’s weights, already sharded for this rank.
Fields§
§q_a_proj: DevicePtr§q_a_layernorm: DevicePtr§q_absorb: DevicePtr[local_heads * kv_lora_rank, q_lora_rank] BF16 — q_b_proj absorbed through
kv_b_proj’s K half, so Q arrives in latent space. See the module header.
kv_a_proj: DevicePtr§kv_a_layernorm: DevicePtr§o_absorb: DevicePtr[hidden, local_heads * kv_lora_rank] BF16, row-parallel — all-reduced by the caller.
🪤 Absorbed, not the raw checkpoint o_proj: the decode kernel leaves its output
in the 512-dim LATENT space, so the projection carries kv_b_proj’s V half folded in.
The raw weight is local_heads * v_head_dim wide — half of this — and feeding the
latent to it reads 2x past every row rather than merely computing the wrong thing.
wk: DevicePtr§k_norm_weight: DevicePtr§k_norm_bias: DevicePtr🪤 REQUIRED. k_norm is a LayerNorm; a .weight-only bind silently drops the
mean subtraction and the bias.
compress_gate: DevicePtr§wq_b: DevicePtr§weights_proj: DevicePtr🪤 Pre-multiplied by index_heads^-0.5 at load — dsa_index_scores does not scale.
ape: DevicePtr[index_kpool, index_head_dim] FP32. 🪤 BF16 on disk; upconverted at load.