Struct NgramRowCache 

pub struct NgramRowCache {
    pub hits: u64,
    pub misses: u64,
    pub evictions: u64,
    /* private fields */
}

One table’s on-NVMe backing file plus its resident row cache.

Fields

hits: u64

misses: u64

evictions: u64

Implementations

impl NgramRowCache

pub fn open( path: &Path, scale_path: Option<&Path>, rows_total: u64, row_stride: usize, slots: usize, ) -> Result<Self>

Open path as the backing store for a table of rows_total rows of row_stride bytes, caching slots of them in pinned GPU-addressable memory. scale_path supplies the per-row f32 scales of an FP8 table.

pub fn open_at( path: &Path, base_offset: u64, scale_path: Option<&Path>, rows_total: u64, row_stride: usize, slots: usize, ) -> Result<Self>

As Self::open, but the table starts at base_offset inside the file — the safetensors-shard case (data_offsets[0] + the header length), which needs no re-save of the checkpoint.

pub fn open_segmented( shards: &[(PathBuf, u64)], rows_per_shard: u64, scale_path: Option<&Path>, row_stride: usize, slots: usize, ) -> Result<Self>

As Self::open_at, but for a table split across equal-sized shards at SCATTERED file offsets — Qwen3.8-Flash-Next’s PLE table, whose 128 shard tensors are not laid out consecutively inside the safetensors file. bases[i] is shard i’s first row; every shard holds rows_per_shard rows.

pub fn table_dev_va(&self) -> Result<u64>

Device VA of the cache’s row table — the embed_table argument of the gather kernels, which then index it by SLOT.

pub fn set_constant_scale(&mut self, scale: f32) -> Result<()>

Give every slot the same scale, for an FP8 table quantized with ONE factor rather than per row.

Filled once here instead of faulted per row: the value does not depend on which row landed in the slot, so a per-fault read would be the same four bytes fetched again for every miss. The gather kernel is the FP8 one either way — it multiplies by scales[slot] and does not care where that came from.

Errors

If the scale arena cannot be allocated.

pub fn scale_dev_va(&self) -> Result<Option<u64>>

Device VA of the [slots] f32 scale array (FP8 tables only).

pub fn stats(&self) -> (u64, u64, u64)

pub fn resolve( &mut self, row_ids: &[u64], out_slots: &mut Vec<u32>, ) -> Result<()>

Resolve row_ids to slot indices, faulting misses in from NVMe.

Every returned slot is PINNED for the caller’s batch: the gather runs after this returns, so a later resolve in the same batch must not evict a row the kernel is about to read. Call Self::end_batch once the gather has been issued.

pub fn end_batch(&mut self)

Release the batch’s pins (call after the gather kernels are issued).