Backend Matrix

Sipp local inference is built on llama.cpp and ggml. Sipp owns the client APIs, endpoint model, scheduling, package bindings, browser lifecycle, and gateway integration; llama.cpp and ggml provide the GGUF runtime and backend kernels.

Backend support therefore has two layers:

• Sipp support: which backend names each package can select and how the backend is built or chosen.
• ggml support: which tensor operations each ggml backend implements.

For the ggml operation-level matrix, use the upstream llama.cpp GGML operations table. That table is generated from llama.cpp backend probes and is the source of truth for per-operation support.

Sipp Backend Names

Upstream llama.cpp/ggml supports more backend families than Sipp currently exposes as package/runtime selectors, including BLAS, CANN, OpenCL, SYCL, ZenDNN, and zDNN. Those appear in the upstream operation matrix but are not first-party Sipp backend names at this time.

Package And Runtime Selection

auto is a runtime selection policy. all is a build/test selector used by sipp and cargo xtask; it builds or checks the host-supported backend set for that target and is not a runtime backend name.

Mixing Backends

Keep build artifact selection separate from engine backend selection.

• A build artifact decides which ggml GPU backends are compiled and loadable in the current process. A CUDA-only artifact does not make Vulkan available, and a Metal-only artifact does not make CUDA or Vulkan available.
• cpu is the exception in the engine policy. When an engine is explicitly planned for cpu, Sipp disables GPU layers, device placement, GPU K/V offload, op offload, flash attention, and GPU residency leasing for that load.
• Explicit GPU selections such as cuda, metal, vulkan, and webgpu must be both compiled into the active artifact and available on the host.
• Node.js and Python choose the native binding at process load with SIPP_NODE_BACKEND or SIPP_PYTHON_BACKEND. Their local model descriptors do not carry a separate per-engine backend field, so use a different process or artifact when you need a different GPU backend.
• Gateway, CLI, browser, and lower-level Rust lifecycle paths expose backend selectors at the target/load/run layer. They can select only from the backend set available to that artifact and host.

Practical examples:

CLI examples:

# Build a CUDA-capable CLI artifact.
sipp build cli --backend cuda

# Use CUDA when the CUDA device is available.
sipp ./models/model.gguf "Explain this model." --chat --backend cuda

# Force CPU for a run; this disables GPU offload for that engine.
sipp ./models/model.gguf "Explain this model." --chat --backend cpu

# This requires a Vulkan-capable artifact; a CUDA-only artifact is not enough.
sipp ./models/model.gguf "Explain this model." --chat --backend vulkan

Gateway target examples:

# Same gateway process, different local targets.
# Each GPU backend must be compiled into the active gateway artifact.
[[targets]]
name = "local-cuda"
type = "local"
model = "./models/model.gguf"
backend = "cuda"

[[targets]]
name = "local-cpu"
type = "local"
model = "./models/model.gguf"
backend = "cpu"

Browser examples:

// Browser local supports CPU and WebGPU backend selection per local endpoint.
await client.add('local-webgpu', {
  kind: 'local',
  model: './models/model.gguf',
  options: { backend: 'webgpu' },
});

await client.add('local-cpu', {
  kind: 'local',
  model: './models/model.gguf',
  options: { backend: 'cpu' },
});

Node.js and Python examples:

# PowerShell: choose the native binding before starting the process.
$env:SIPP_NODE_BACKEND = "cuda"
node .\examples\node\chat.mjs .\models\model.gguf "Explain this model."

$env:SIPP_NODE_BACKEND = "cpu"
node .\examples\node\chat.mjs .\models\model.gguf "Explain this model."

# Bash: choose the native binding before starting the process.
SIPP_PYTHON_BACKEND=cuda \
  python examples/python/chat.py ./models/model.gguf "Explain this model."

SIPP_PYTHON_BACKEND=cpu \
  python examples/python/chat.py ./models/model.gguf "Explain this model."

Build Matrix

sipp build all is intentionally conservative. Use an explicit backend build when you need CUDA, Metal, or Vulkan artifacts.

Operation Support

ggml backends do not all implement the same operation set. Common transformer inference paths are covered by the backends Sipp exposes, but support for a specific model family depends on the ggml operations used by that model and the selected backend.

Use these rules when diagnosing backend issues:

• If a model works on cpu but fails on a GPU backend, check the upstream ggml operations matrix for the missing operation.
• If a GPU backend lacks an operation, llama.cpp/ggml may fall back for some paths, keep tensors on CPU for that operation, or fail depending on the graph and backend policy.
• If a package cannot see a backend at runtime, check that the artifact was built or installed for that backend and that the device driver/runtime is visible to the process.
• Browser webgpu depends on both compiled WebGPU support and browser adapter availability. Use backend: 'cpu' to force the browser CPU path.

For local verification from a source checkout:

sipp doctor --target node --backend vulkan
sipp run llama backend-ops --backend vulkan --mode support
sipp run llama backend-ops --backend cuda --mode perf --op MUL_MAT

The llama backend-ops command builds llama.cpp’s backend operation tool for the selected backend and is useful when investigating operation coverage or performance outside the Sipp client path.

Practical Selection

Use cpu first when validating a model or reproducing correctness issues. Move to a GPU backend after the model, prompt format, and runtime config are known to work.

Use webgpu for browser-local acceleration when the application can require a modern WebGPU browser. Keep a CPU fallback for browsers, drivers, and devices that do not expose a compatible adapter.

Use cuda for NVIDIA-heavy native deployments and metal for Apple Silicon or tested AMD macOS deployments. On Intel Macs with integrated GPUs, use cpu unless the exact model, context size, and device have been tested and Metal is stable and faster than CPU. Use vulkan when you want a cross-vendor native GPU path and have tested the target driver stack. On macOS, prefer Metal over Vulkan unless you are specifically testing LunarG’s Vulkan-over-Metal drivers.