Providers

Universal LLM Client supports multiple providers through a single API. Each provider has specific behaviors for structured output, streaming, and feature support.

Provider Support Matrix

Feature	OpenAI (compat)	Google/Vertex	Ollama	Anthropic	LlamaCpp
Chat	✅	✅	✅	✅	✅
Streaming	✅	✅	✅	✅	✅
Structured Output	✅	✅	✅	✅	✅
Tool Calling	✅	✅	✅	✅	✅
Vision/Images	✅	✅	✅	✅	❌
Embeddings	✅	✅	✅	❌	✅
Strict JSON Mode	✅ (native)	❌ (stripped)	❌	⚠️ (prompted)	❌
Prompt Caching	Provider-dependent	Limited	N/A	✅ (strong)	N/A
Native Thinking	Via decoder / fields	✅ (thinkingConfig)	✅ (think)	✅ (extended + signatures)	Via decoder

OpenAI

Supports OpenAI and any OpenAI-compatible API (OpenRouter, Groq, Together AI, LM Studio, vLLM, etc.).

const model = new AIModel({
  model: 'gpt-4o-mini',
  providers: [
    { type: 'openai', apiKey: process.env.OPENAI_API_KEY },
  ],
});

Structured Output

OpenAI uses response_format: { type: 'json_schema' } natively. The library sends your Zod schema as a JSON Schema in the request, and OpenAI constrains its output accordingly.

Strict mode is enabled by default — OpenAI guarantees the response matches the schema exactly:

// Strict mode (default)
const result = await model.generateStructured(MySchema, messages);

// Disable strict for compatible endpoints that don't support it
const result = await model.generateStructured(MySchema, messages, {
  output: { strict: false },
});

Compatible Services

For OpenAI-compatible endpoints, just set the url:

{ type: 'openai', url: 'https://openrouter.ai/api/v1', apiKey: '...' }
{ type: 'openai', url: 'http://localhost:1234/v1', apiKey: 'lm-studio' }
{ type: 'openai', url: 'http://localhost:8000/v1', apiKey: '...' } // vLLM

Advanced transport flexibility (new in this release):

You can now control auth, query parameters, and base path without custom code:

// Azure OpenAI (recommended pattern)
{
  type: 'openai',
  url: 'https://YOUR-RESOURCE.openai.azure.com/openai/deployments/YOUR-DEPLOYMENT',
  apiBasePath: '',                    // do not append /v1
  queryParams: { 'api-version': '2024-10-21' },
  headers: { 'api-key': process.env.AZURE_OPENAI_KEY },
  // No need to put the query in the url string anymore
}

// Custom gateway with api-key style auth (no "Bearer")
{
  type: 'openai',
  url: 'https://my-gateway.example.com',
  authHeader: 'api-key',
  authPrefix: '',
  apiKey: process.env.GATEWAY_KEY,
  queryParams: { 'region': 'us-east' },
}

// Extra arbitrary headers (merged after auth)
{
  type: 'openai',
  url: '...',
  headers: { 'x-custom': 'value', 'api-key': '...' }, // OpenAI-compatible & Ollama (via buildHeaders)
}

These options are typed on every ProviderConfig, but are honored by the OpenAI-compatible provider — with headers, authHeader, and authPrefix also applied by Ollama (both use buildHeaders). queryParams and apiBasePath are OpenAI-compatible only. Google/Vertex and Anthropic build their own auth headers and URLs and ignore these knobs.

For Anthropic-specific prompt caching:

await model.chat(messages, { enablePromptCaching: true, maxTokens: 8192 });

This causes the client to mark the system prompt with Anthropic's cache_control: { type: "ephemeral" }. See the Anthropic section for details.

Google / Vertex AI

Supports Google's Generative AI API and Vertex AI.

// Google AI Studio
const model = new AIModel({
  model: 'gemini-2.0-flash',
  providers: [
    { type: 'google', apiKey: process.env.GOOGLE_API_KEY },
  ],
});

// Vertex AI
const model = new AIModel({
  model: 'gemini-2.0-flash',
  providers: [
    {
      type: 'vertex',
      apiKey: process.env.VERTEX_API_KEY,
      region: 'us-central1',
    },
  ],
});

Structured Output

Google uses responseMimeType: 'application/json' with a responseSchema. The library automatically strips JSON Schema features that Google doesn't support:

• pattern
• minLength / maxLength
• minimum / maximum / exclusiveMinimum / exclusiveMaximum
• additionalProperties

INFO

Your Zod schema can use .min(), .max(), .regex(), etc. freely — the library strips unsupported features from the schema sent to Google while keeping them for validation of the response.

Ollama

Local LLM provider using Ollama's REST API.

const model = new AIModel({
  model: 'llama3.2',
  providers: [
    { type: 'ollama', url: 'http://localhost:11434' },
  ],
});

Structured Output

Ollama accepts a format parameter with the JSON Schema object directly. The library converts your Zod schema and sends it as the format:

const result = await model.generateStructured(MySchema, [
  { role: 'user', content: 'Extract entities from...' },
]);

TIP

For best results with structured output on Ollama, use models that support JSON mode well (e.g., llama3.2, mistral, qwen2.5).

LlamaCpp

Direct connection to a llama.cpp server.

const model = new AIModel({
  model: 'default',
  providers: [
    { type: 'llamacpp', url: 'http://localhost:8080' },
  ],
});

Uses the OpenAI-compatible API format internally.

Anthropic (Claude)

Native support for Anthropic's Messages API (distinct from OpenAI wire format).

const model = new AIModel({
  model: 'claude-sonnet-4-20250514',
  providers: [
    { type: 'anthropic', apiKey: process.env.ANTHROPIC_API_KEY },
  ],
});

Key Differences Handled by the Library

• Content blocks (text, tool_use, tool_result, thinking, image) instead of flat messages.
• system is a top-level field (extracted from role: 'system' messages).
• Tool results are placed inside user messages as tool_result blocks.
• Streaming uses typed content_block_* events (including thinking_delta and signature_delta).
• Strong prompt caching and extended thinking support.

Prompt Caching

const response = await model.chat(messages, {
  enablePromptCaching: true,
  // ... other options
});

When enabled, the client sets a system-prefix breakpoint, checkpoints the last two assistant messages, and sets top-level cache_control: { type: 'ephemeral' }. Anthropic automatically advances the latter over the growing conversation. The assistant checkpoints preserve history reuse when a harness replaces temporary context at the end of each request. These use at most four cache slots altogether. The default cache lifetime is five minutes; model-specific minimum prompt sizes still apply. Passing false omits all markers.

usage.inputTokens includes uncached input, cache reads and cache writes. usage.cachedTokens reports reads, and usage.cacheWriteTokens reports writes; both are already included in inputTokens. Missing counters remain absent, rather than being reported as measured misses. Streaming counters are cumulative snapshots. See Anthropic's prompt caching guide.

For the Responses transport, ChatOptions.conversationId supplies the conversation routing headers for one request. It overrides the constructor default without mutating the shared client, so concurrent chats retain separate routing values. This is distinct from sessionKey, which only controls local router affinity.

Structured Output & Thinking

Recent Claude 4.x models have improving native schema enforcement. The library always performs final validation. Thinking / reasoning is surfaced via the standard reasoning field and DecodedEvent stream events.

Embeddings

Anthropic does not offer embeddings — use a different provider in your failover chain if you need them.

Most Other Providers: Use the OpenAI-Compatible Path

The vast majority of inference providers (hosted and self-hosted) speak the OpenAI Chat Completions format (or a close superset). You do not need a dedicated adapter.

Just use { type: 'openai', url: '...' }:

// xAI Grok
{ type: 'openai', url: 'https://api.x.ai/v1', apiKey: process.env.XAI_API_KEY }

// Mistral
{ type: 'openai', url: 'https://api.mistral.ai/v1', apiKey: process.env.MISTRAL_API_KEY }

// DeepSeek (very low cost)
{ type: 'openai', url: 'https://api.deepseek.com', apiKey: process.env.DEEPSEEK_API_KEY }

// Cohere Compatibility API
{ type: 'openai', url: 'https://api.cohere.ai/compatibility/v1', apiKey: process.env.COHERE_API_KEY } // verify exact path in Cohere docs

// Groq (fast)
{ type: 'openai', url: 'https://api.groq.com/openai/v1', apiKey: process.env.GROQ_API_KEY }

// Together AI
{ type: 'openai', url: 'https://api.together.xyz/v1', apiKey: process.env.TOGETHER_API_KEY }

// Fireworks
{ type: 'openai', url: 'https://api.fireworks.ai/inference/v1', apiKey: process.env.FIREWORKS_API_KEY }

// OpenRouter (aggregator)
{ type: 'openai', url: 'https://openrouter.ai/api/v1', apiKey: process.env.OPENROUTER_API_KEY }

// Perplexity Sonar (grounded)
{ type: 'openai', url: 'https://api.perplexity.ai', apiKey: process.env.PERPLEXITY_API_KEY }

// Self-hosted vLLM / TGI / LM Studio / etc.
{ type: 'openai', url: 'http://localhost:8000/v1', apiKey: 'not-needed-for-local' }

Notes for Compatible Endpoints

• Many do not implement OpenAI's full strict JSON schema mode → pass output: { strict: false } when using generateStructured / chat({ output }).
• Tool streaming and parallel calls are widely supported but accumulation logic can be quirky; the library normalizes IDs and empty args.
• Usage / cost headers vary; the auditor still receives normalized TokenUsageInfo.
• For Azure OpenAI: the URL structure (/deployments/{deployment}/chat/completions?api-version=...) and api-key header are different. You can often construct the full URL + pass custom headers (see advanced config) or request an azure provider type for first-class ergonomics.

See the research document docs/research/provider-api-landscape-2026.md for a detailed 2026 survey of wire formats, why only Anthropic + Google warrant native clients, Bedrock guidance, Responses API notes, and more.

Provider Failover

When using multiple providers, the library automatically fails over to the next provider on errors:

const model = new AIModel({
  model: 'gpt-4o-mini',
  providers: [
    { type: 'openai', apiKey: process.env.OPENAI_API_KEY, priority: 0 },
    { type: 'google', apiKey: process.env.GOOGLE_API_KEY, priority: 1 },
    { type: 'ollama', url: 'http://localhost:11434', priority: 2 },
  ],
  retries: 2, // retries per provider before moving to next
});

How failover works with structured output

1. The schema is sent in the provider's native format (JSON Schema for OpenAI, responseSchema for Google, format for Ollama)
2. If a provider fails, the next provider receives the same schema re-formatted for its API
3. Response validation is centralized in the Router — regardless of which provider responds, the same Zod validation runs

Health Tracking

Each provider has health status with automatic cooldowns:

const statuses = model.getProviderStatus();
// [
//   { id: 'openai-0', healthy: true, errorCount: 0 },
//   { id: 'google-1', healthy: false, errorCount: 3, coolingDown: true },
// ]