Library API Overview

The Sipp libraries for Rust, Node.js, Python, and Browser expose the same endpoint-oriented client model.

At a high level:

1. Register an endpoint with add.
2. Keep the returned EndpointRef.
3. Pass that reference to query, chat, or embed.

This keeps application code the same whether inference runs locally, through a gateway, through a provider, or across a hybrid setup.

Core Client Methods

SippClient exposes four primary methods:

add() — Register an Endpoint

add(id: string, descriptor: EndpointDescriptor) -> EndpointRef

add registers an endpoint with the current client instance.

The id is caller-defined and scoped to the client. Reusing an id replaces the existing endpoint. The returned EndpointRef is a lightweight handle with:

Pass the returned EndpointRef to query, chat, or embed to choose where the operation runs.

Local Endpoint

A local endpoint loads a GGUF model into the current process. The application owns model selection, runtime lifecycle, and cleanup.

Use a local endpoint when the current process should own model execution.

Gateway Endpoint

A gateway endpoint sends requests to a remote Sipp gateway over HTTP. The gateway process owns provider credentials, local model paths, access policy, concurrency, and metrics.

Use a gateway endpoint when a separate service should own model access and operational policy.

Provider Endpoint

A provider endpoint calls a model provider directly. This is intended for trusted server-side code that manages its own credential lifecycle.

Use a provider endpoint when server-side code should call a provider API directly without a Sipp gateway.

query() — Generate from a Raw Prompt

query(request: SippQueryRequest) -> SippTextRun

query sends the prompt string to the selected endpoint exactly as supplied. No chat template is applied.

Use query when the application owns the full prompt shape, including custom templates, completion-style models, encoder-decoder text models, few-shot prompts, or agent loops that render prompts themselves.

Request Fields

Return Value

query returns a SippTextRun.

SippTextResponse contains the generated text, finishReason, token usage, and optional localStats for local endpoints.

chat() — Generate from Role Messages

chat(request: SippChatRequest) -> SippTextRun

chat sends ordered role/content messages to the selected endpoint. The endpoint owns message rendering.

Request Fields

Return Value

chat returns the same SippTextRun shape as query.

embed() — Generate an Embedding

embed(request: SippEmbedRequest) -> SippEmbeddingRun

embed produces a single embedding vector from text input. It does not accept generation options and does not stream tokens.

Request Fields

Return Value

embed returns a SippEmbeddingRun.

SippEmbeddingResponse contains the float values array, optional token usage, the pooling strategy, and the normalized flag.

Gateway and Client Symmetry

The same SippClient API works on both sides of the gateway boundary.

Server Side

A server process creates a SippClient, registers local endpoints, and maps HTTP routes to query, chat, or embed.

Server client:
  add("local-model", LocalDescriptor { modelPath, config })
  -> route handler decodes HTTP request
  -> route handler calls client.query/chat/embed
  -> route handler encodes HTTP response

The first-party Gateway Server uses this pattern. Application-owned Node, Python, or Rust servers can also use it through the gateway profile helpers.

Client Side

A client process creates a SippClient, registers gateway endpoints, and calls query, chat, or embed the same way it would call a local endpoint.

Client client:
  add("remote", GatewayDescriptor { target, baseUrl, authentication })
  -> client.query/chat/embed({ endpoint: ref, ... })
  -> request is sent to the gateway over HTTP

Hybrid Pattern

A single client can register multiple endpoint kinds. The application chooses where an operation runs by passing a different endpoint reference.

localRef = client.add("local", LocalDescriptor { ... })
gatewayRef = client.add("gateway", GatewayDescriptor { ... })

client.query({ endpoint: localRef, prompt, ... })
client.query({ endpoint: gatewayRef, prompt, ... })

The operation code stays the same. Only the endpoint reference changes.

Why the Endpoint Model Matters

The endpoint model gives applications one API surface across multiple deployment shapes.

Visual Summary

flowchart LR
    %% -------------------------
    %% Node Styling
    %% -------------------------
    classDef client_node fill:#eef6ff,stroke:#4a90e2,stroke-width:1.5px,color:#111,rx:6,ry:6;
    classDef setup_node fill:#f7f7f7,stroke:#999,stroke-width:1px,color:#111,rx:6,ry:6;
    classDef runtime_node fill:#f3fff0,stroke:#52a852,stroke-width:2px,color:#111,rx:6,ry:6;
    classDef gateway_node fill:#fff7e6,stroke:#d99000,stroke-width:2px,color:#111,rx:6,ry:6;
    classDef provider_node fill:#f8f0ff,stroke:#8e44ad,stroke-width:1.5px,color:#111,rx:6,ry:6;

    %% -------------------------
    %% Client Process
    %% -------------------------
    subgraph CLIENT["Client Process"]
        direction TB
        CApp["Application Code"]:::client_node
        CClient["SippClient<br/>add(...) -> EndpointRef<br/>query / chat / embed"]:::client_node
        CApp --> CClient

        %% Logical grouping for endpoint registration options
        subgraph CSetup["Endpoint Setup (options)"]
            direction LR
            CLocalEP["local (GGUF)"]:::setup_node
            CGatewayEP["gateway (Remote)"]:::setup_node
            CProviderEP["provider (API)"]:::setup_node
        end
        CClient -. "Registers" .-> CSetup

        %% Local execution flow for local ref
        subgraph CLocalRuntime["Local Runtime"]
            direction LR
            CLocalRun["GGUF Runtime"]:::runtime_node
        end

        %% Connection for local usage
        CClient -- "Local Ref (query)" --> CLocalRun
    end

    %% -------------------------
    %% Server Process
    %% -------------------------
    subgraph SERVER["Server Process / Gateway Server"]
        direction TB
        SGateway["Gateway Server<br/>HTTP: /v1/query, /chat, /embed"]:::gateway_node
        SClient["SippClient (same lib)"]:::client_node
        SGateway --> SClient

        %% Logical grouping for endpoint registration options
        subgraph SSetup["Endpoint Setup (options)"]
            direction LR
            SLocalEP["local (GGUF)"]:::setup_node
            SProviderEP["provider (API)"]:::setup_node
        end
        SClient -. "Registers" .-> SSetup

        %% Local execution flow for local ref
        subgraph SLocalRuntime["Local Runtime"]
            direction LR
            SLocalRun["GGUF Runtime"]:::runtime_node
        end

        %% Connection for local usage
        SClient -- "Local Ref (query)" --> SLocalRun
    end

    %% -------------------------
    %% External Providers
    %% -------------------------
    Providers["Provider APIs<br/>OpenAI / Gemini / Anthropic / etc."]:::provider_node

    %% -------------------------
    %% Cross-process / Remote connections
    %% -------------------------
    CClient == "Gateway Ref (query)" ==> SGateway
    CClient == "Provider Ref (query)" ==> Providers
    SClient == "Provider Ref (query)" ==> Providers

    %% -------------------------
    %% Styling Assignment to Nodes
    %% -------------------------
    class CApp,CClient client_node;
    class CLocalEP,CGatewayEP,CProviderEP,SLocalEP,SProviderEP setup_node;
    class CLocalRun,SLocalRun runtime_node;
    class SGateway gateway_node;
    class Providers provider_node;

Related Docs

• Using the Core Library — per-language install steps and examples.
• Inference Operations — operation contracts, template behavior, and gateway target mapping.
• Local Inference — model sources, runtime options, threads, and browser execution.
• Gateway and Hybrid Inference — deployment shapes, endpoint model, and authentication patterns.
• Runtime Options — complete option layer map and field reference.