AI Gateway Setup and API Guide
Overview
The AI Gateway provides a centralized way to connect users and applications to different AI providers, such as OpenAI, Anthropic, AWS Bedrock, and custom AI services.
Instead of connecting each application directly to an AI provider, requests are sent through the AI Gateway. The gateway then handles provider routing, authentication, firewall inspection, auditing, and usage tracking.
Main Capabilities
The AI Gateway supports:
Centralized access to multiple AI providers
Provider and model management
Account-level and user-level authentication settings
AI firewall and policy enforcement
Usage auditing and cost tracking
OpenAI-compatible API requests
1. Admin Center Setup
Managing AI Providers
Administrators manage providers from the AI Gateway Settings page in the Admin Center. A provider can be enabled or disabled, tested, edited, or removed.
Provider Configuration
Configure the provider display name, base URL, supported API standards, authentication, model synchronization, and enabled state.
Configuration | Current behavior |
|---|---|
Provider integrations | OpenAI-compatible providers, xAI, Azure Foundry, Anthropic, AWS Bedrock, AWS Bedrock Mantle, Google Gemini, and Google Vertex. |
API standards | v1/chat/completions, v1/responses, and v1/messages. The default is v1/chat/completions. |
Authentication | Provider-specific API key, bearer token, custom header, Anthropic, Azure Foundry, Google, AWS, OAuth, or no-auth configuration where supported. |
Base URL | The upstream provider URL used by the gateway. Provider adapters add the protocol-specific route. |
Model synchronization | Enable or disable synchronization of all models for the provider. Models can also be managed manually. |
Use the provider test action after saving credentials and the base URL. The provider API standard must match the endpoint contract that the provider supports.
2. Model Management
Models are managed under their provider. Choose automatic synchronization when the provider exposes a compatible model-list endpoint, or use manual addition when discovery is unavailable or you want to curate the available models.
Automatic model discovery and synchronization
Configure and save the provider credentials, base URL, API standard, and enabled state.
In the provider configuration, enable Sync All for model management.
In the Models section, set the AI Gateway Model Sync Interval (hours) and click Save. Use Sync Now when the catalog must be refreshed immediately.
Review the discovered models, enable the models that should be available, and use Test Model to verify them.
Automatic synchronization adds or updates models returned by the provider's compatible model-list endpoint. The default synchronization interval is 24 hours.
Manual model addition
Use manual management when the provider does not expose a compatible model-list endpoint, when the model is a private deployment or alias, or when only selected models should be available.
Configure and save the provider, leaving model management set to Manual.
In the Models section, select the provider and click Add Model.
Enter the exact provider model identifier used in upstream requests, together with the display name and any optional description or metadata, then save.
Enable the model and use Test Model to verify credentials, routing, and the provider API standard. Configure fallback models if required.
For either method, the provider and model must be enabled before normal gateway selection. The model identifier should match the provider's identifier exactly; clients can verify the available identifiers through Public AI Gateway API, including the /v1/models endpoint.
For the current controls, see AI Gateway - Providers & Models.
3. User-Level Authentication and Gateway API Keys
The gateway separates administrator-managed provider credentials from user-specific credentials. Users can store a personal provider authentication override for their own usage; saving or deleting that override does not change the global provider configuration.
Gateway API Keys
Administrators manage gateway API keys from the AI Gateway API Keys page. Keys can be managed for human users and for non-human identities used by applications, agents, scripts, services, and other machine-to-machine integrations.
Every non-human identity has a responsible human user or group owner. A non-human identity cannot own another non-human identity. The API key headers and authentication examples are documented on the Public AI Gateway API page.
4. Public AI Gateway API
The standalone public API documentation is maintained on the Public AI Gateway API page.
That page documents /v1/chat/completions, /v1/responses, /v1/messages, /v1/models, authentication headers, request and response formats, inline attachments, provider compatibility notes, and examples.
5. AI Gateway Firewall and Policies
Firewall protection and AI Gateway policy enforcement are related but distinct controls. Account-level firewall settings determine which interactions are audited or inspected, while AI Gateway settings and policies determine how standalone gateway requests are inspected and governed.
Account Firewall Settings
Setting | Default / options | Purpose |
|---|---|---|
FirewallEnabled | Yes | Enables account firewall protection. |
RealTimeFirewallAuditingScope | Prompt and Response | Controls which interaction parts are written to real-time firewall auditing. |
RealTimeFirewallInspectionScope | Prompt | Controls which interaction parts are inspected in real time. |
RealTimeFirewallInspectionAction | Block; also None, Block and Flag, Mask, Block and Mask, or Block, Flag and Mask | Controls which policy actions are applied inline. |
FirewallClassificationSource | Nexus | Selects Nexus or AI Gateway as the firewall classification source. |
AI Gateway classification provider/model | Required when the source is AI Gateway | Selects the provider and model used for AI Gateway firewall classification. |
Firewall setting changes may take up to five minutes to take effect because of caching and performance optimizations.
AI Gateway Defaults and Policy Overrides
AIGatewayRealTimeInspection controls whether AI Gateway requests are inspected in real time by default. The documented account default is Yes.
AIGatewayRealTimeToolInspection controls tool-call inspection. Its account default is Audit Only; the available modes are Disabled, Audit Only, and Realtime. This setting is rendered through AI Gateway Policies rather than the generic Account Settings page.
An individual AI Gateway policy can override the default prompt-inspection and tool-inspection behavior.
AI Gateway Policy Types
The AI Gateway Policies page supports two policy types:
Model access: allow or block requests, optionally targeting an agent, user or group, provider, and model. Allow rules can use Any, Preset, or Optimize behavior.
Usage control: limit usage by Cost, Tokens, or Requests for a configured period, with a Block, Throttle, or Allow limit action and optional notifications.
Runtime Inspection Behavior
For a request that requires prompt inspection, the gateway evaluates the configured prompt and policy controls before forwarding the request to the selected provider. A request may be blocked, flagged, or masked according to the effective inspection action and matched policy. Prompt-blocked responses preserve the endpoint protocol so the client can keep the conversation history, while the blocked event is recorded for auditing.
Tool-call inspection is separate from prompt inspection. Audit Only evaluates tool calls for auditing without enforcement; Realtime evaluates them during the request and can block a tool call; Disabled turns off tool-call policy auditing.
Important: real-time auditing of responses (when enabled in the account auditing scope) is not the same as general provider-response firewall blocking. The standalone gateway's inline firewall controls documented here are prompt inspection and tool-call inspection.
6. Usage Auditing and Cost Tracking
The standalone gateway persists request auditing and usage information when the required database and integration settings are available. Audit records can include the authenticated user or identity, provider, model, request status, policy information, request correlation identifiers, agent/application metadata, and the gateway answer.
When the provider returns usage data, the gateway records prompt, completion, total, and cache-related token values and calculates the request cost from the configured model pricing. Usage information is also used to enforce AI Gateway usage-control policies and to support reporting in AI Gateway Auditing and AI Usage Insights.
Applications should send gateway metadata such as ApplicationName, AgentName, or ProjectName when applicable so usage can be correlated with the originating integration. The gateway removes these routing metadata fields before sending the provider request.
7. Troubleshooting
Request is blocked
Review the account firewall settings, effective AI Gateway policy, prompt-inspection mode, tool-inspection mode, and the selected provider/model. A policy can be inherited from the account default or explicitly override it.
Model is not available
Check that the provider and model are enabled, that the model is available to the authenticated account, and that the requested model name matches the model catalog. If a provider suffix is required, use the model identifier shown by /v1/models.
Authentication fails
For the gateway, use an API key managed in AI Gateway API Keys and send it in the documented gateway header.
For the upstream provider, verify the provider auth type, key or token, custom header, base URL, and provider-specific fields.
If a user override is configured, verify that the personal credentials are still valid.
Provider does not return models
Model discovery requires a compatible provider model-list endpoint. Add the models manually when discovery is unavailable, then enable and test them from AI Gateway Settings.
Model pricing configuration
Configure shared site defaults and account-level pricing overrides on the AI Gateway Model Pricing and Cost Configuration page.