com.microsoft.PagedAttention

com.microsoft · ONNX Runtime contrib operator · contrib since_version 1

Description

Attention over a block-based (paged) KV cache: cumulative_sequence_length marks the sequence boundaries and block_table maps a sequence's history onto scattered blocks. This step's K/V are scattered into the cache, then attended with that history. Grouped-query heads, scale, packed [Q|K|V], slot_mapping, and float16 cache storage are supported; the cache outputs alias the input caches and are updated in place. Rotary embeddings, softcap, local windows, LATENT layout, narrower value heads, quantized KV, head sinks, q/k normalization, scales, and attention metadata are not implemented.

See the ONNX Runtime PagedAttention contrib-operator spec for the reference semantics.

Inputs

Outputs

Attributes

Attributes and default values (overridable per request):

Type constraints

Implementation variants

One implementation is selected per call from the device capabilities, the request shapes and the dtypes; these notes say what each one covers.

• separate_derived_splitk — Splits each token's key history into contiguous ranges, processes one range per workgroup, and merges the resulting online-softmax states. This adds parallel key ranges for grouped-query decode shapes with few (token, KV head) tuples.
• separate_slot_splitk — Splits each token's key history into contiguous ranges, processes one range per workgroup, and merges the resulting online-softmax states. This adds parallel key ranges for grouped-query decode shapes with few (token, KV head) tuples.
• packed_derived_splitk — Splits each token's key history into contiguous ranges, processes one range per workgroup, and merges the resulting online-softmax states. This adds parallel key ranges for grouped-query decode shapes with few (token, KV head) tuples.
• packed_slot_splitk — Splits each token's key history into contiguous ranges, processes one range per workgroup, and merges the resulting online-softmax states. This adds parallel key ranges for grouped-query decode shapes with few (token, KV head) tuples.

Device requirements

Every implementation variant requires shader-f16; the package has no variant-level fallback without that capability.

Files

• metadata.json — kernel metadata (id, digests, per-variant templates, provenance)
• manifest.json — the op contract (source of truth)
• test.json — correctness cases
• bench.json — benchmark + tuning cases
• attn-flash-decode-splitk-merge.wgsl.jinja
• paged-attention.wgsl.jinja
• paged-scatter-kv.wgsl.jinja

Use with @huggingface/kernels

npm install --save-exact @huggingface/kernels@0.0.1-preview.2

Required output shapes and logical data types are inferred from the supplied inputs and attributes; result tensors are allocated automatically.

The version: 1 option selects the published kernel contract; it is independent of any operator opset, contrib since_version, or model version. It follows the v1 branch as fixes land. To pin exact artifact bytes, pass a 40-character commit revision instead of version.

Replace each *Data placeholder with a typed array containing the corresponding input data.

import { getKernel } from "@huggingface/kernels";

const kernel = await getKernel("webgpu-kernels/com.microsoft.PagedAttention", { version: 1 });
const { keyCacheT, valueCacheT, outputT } = await kernel({
  queryT: { data: queryTData, shape: [2, 4] },
  keyT: { data: keyTData, shape: [2, 2] },
  valueT: { data: valueTData, shape: [2, 2] },
  keyCacheT: { data: keyCacheTData, shape: [3, 2, 1, 2] },
  valueCacheT: { data: valueCacheTData, shape: [3, 2, 1, 2] },
  cumulativeSequenceLengthT: { data: cumulativeSequenceLengthTData, shape: [2] },
  pastSeqlensT: { data: pastSeqlensTData, shape: [1] },
  blockTableT: { data: blockTableTData, shape: [1, 3] },
}, {
  attrs: { num_heads: 2, kv_num_heads: 1 },
});