Struct HighSpeedSwap 

pub struct HighSpeedSwap { /* private fields */ }

Implementations

impl HighSpeedSwap

pub fn offload_block( &mut self, ctx: &CudaCtx, layer: u32, block: u32, k_block_dev: u64, k_block_host: &[bf16], v_block_host: &[bf16], ) -> Result<()>

Persist a freshly-written KV block to disk and update the predictor’s per-block K_lr. K block layout is [block_size, num_kv_heads, head_dim] BF16 in both *_dev (used for projection) and *_host (used for the per-(kv_head) disk stripe).

pub fn offload_block_on_stream( &mut self, stream: u64, layer: u32, block: u32, k_block_dev: u64, k_block_host: &[bf16], v_block_host: &[bf16], ) -> Result<()>

Stream-only variant for production callers (spark-model decode path). stream must already be bound to the current thread’s CUDA context.

pub fn offload_block_no_predict_on_stream( &mut self, stream: u64, layer: u32, block: u32, k_block_host: &[bf16], v_block_host: &[bf16], ) -> Result<()>

FP8/quantized callers: identical to offload_block_on_stream but skips the predictor’s per-block K projection (since k_block_dev is not BF16-laid-out — running the BF16 kernel on it would OOB-read into adjacent blocks). Eviction policy degrades to LRU-only for these blocks; correctness is preserved.

pub fn attend_layer( &mut self, ctx: &CudaCtx, layer: u32, seq_block_ids: &[u32], q_dev: u64, output_dev: u64, ) -> Result<()>

Run streaming attention for one (layer, sequence). q_dev is the full [num_q_heads × head_dim] BF16 query for this step; seq_block_ids is the sequence’s full block list; output_dev receives the [num_q_heads × head_dim] BF16 attention output.

pub fn attend_layer_on_stream( &mut self, stream: u64, layer: u32, seq_block_ids: &[u32], q_dev: u64, output_dev: u64, ) -> Result<()>

Stream-only variant for production callers (spark-model decode path). stream must already be bound to the current thread’s CUDA context.

Backwards-compat: defaults last_block_valid_slots to block_size, i.e. no causal masking — appropriate for decode where the active block’s stale slots are zero-init from zero_block. For prefill, callers MUST use attend_layer_on_stream_with_q_pos to pass the query’s absolute position, otherwise future tokens within the active block leak into past queries.

pub fn attend_layer_on_stream_with_q_pos( &mut self, stream: u64, layer: u32, seq_block_ids: &[u32], q_dev: u64, output_dev: u64, last_block_valid_slots: i32, ) -> Result<()>

Causal-masking variant: last_block_valid_slots controls how many slots of the LAST block in seq_block_ids are consumed by the attention kernel. For prefill query at absolute position q_pos, pass (q_pos % block_size) + 1 to mask out future positions in the active block.

pub fn pool(&self) -> &ScratchPool

Test/diag accessors.

pub fn predictor(&self) -> &Predictor

pub fn config(&self) -> &HighSpeedSwapConfig

impl HighSpeedSwap

pub fn new( ctx: &CudaCtx, cfg: HighSpeedSwapConfig, model: ModelDims, ) -> Result<Self>

pub fn new_on_stream( stream: u64, cfg: HighSpeedSwapConfig, model: ModelDims, ) -> Result<Self>

Stream-only constructor for production callers that already own a CUDA context (spark-model). The provided stream is used only for init-time copies (uploading the projection matrix P); subsequent per-step calls take their own stream argument.

pub fn alloc_disk_block_id(&mut self) -> Option<u32>

pub fn inc_disk_ref(&mut self, id: u32)

pub fn dec_disk_ref(&mut self, id: u32) -> u32

pub fn disk_refcount(&self, id: u32) -> u32

pub fn disk_free_count(&self) -> usize

pub fn diagnostic_summary(&self) -> HighSpeedSwapDiagnostic

Aggregated diagnostic summary across all layers (Phase 6.1.j). Use to log periodic state during long-running decode loops; the scheduler can call this once per N steps to verify HBM-shrink behavior is on track.