Respan Gateway

Respan Gateway: The Ultimate AI Gateway for Production LLM Routing

In the rapidly evolving landscape of artificial intelligence, managing multiple Large Language Models (LLMs) requires a robust infrastructure that ensures reliability, scalability, and cost-efficiency. Respan Gateway emerges as the premier solution for developers and enterprises looking to trace, evaluate, and improve AI agents. As a production-grade AI gateway, it provides a unified interface to interact with over 500 models, offering unparalleled control over LLM routing, failover strategies, and observability.

What is Respan Gateway?

Respan Gateway is a comprehensive production gateway designed to act as a single point of entry for all your LLM calls. Whether you are using OpenAI-style calls or native provider SDKs, the Respan Gateway serves as a unified router or a provider passthrough. It simplifies the complexity of managing multiple API keys and endpoints by providing one stable URL (https://api.respan.ai/api/) for more than 500 models from leading providers like OpenAI, Anthropic, Google Gemini, and Groq.

By routing traffic through the Respan Gateway, every request is automatically logged, allowing teams to gain deep insights into latency, cost, and performance. It is built to bridge the gaps commonly found in production environments, such as lack of failover, key sprawl, and siloed logs.

Key Features of Respan Gateway

Unified API and Model Passthrough

One of the standout features of the Respan Gateway is its ability to route OpenAI-style calls to over 500 different models. Alternatively, developers can maintain each provider's native SDK while still using the Respan Gateway as a passthrough endpoint. This flexibility ensures that every single request—regardless of the provider—is captured and logged for analysis.

Intelligent Failover and Fallback Models

Production uptime is critical. The Respan Gateway allows you to stay operational even when specific models fail or hit rate limits. By setting fallback_models on the request or within the settings, the gateway automatically tries the next model in your list if the primary one errors out. This load balancing across keys ensures your users never experience downtime due to upstream provider issues.

Advanced Response Caching

To reduce both costs and latency, the Respan Gateway includes built-in response caching. Developers can enable cache_enabled to reuse answers for repeat prompts. Critically, it supports cache_by_customer, ensuring that one user’s sensitive data is never returned to another user, solving a common security gap in shared cache setups.

Spend Limits and Cost Management

Managing API costs is made easy with the Respan Gateway. You can set soft warnings or hard caps per API key. When a threshold is crossed, the system can send alerts via Slack or email, preventing unexpected bills and providing clear headroom visibility for every project.

Comprehensive Tracing and Metadata

Every call made through the Respan Gateway is transformed into a detailed trace tree. By adding customer_identifier and custom metadata, teams can filter logs and traces by feature, tenant, or specific threads. This level of granularity is essential for debugging multi-turn traffic and understanding user behavior at scale.

Use Cases for Respan Gateway

Eliminating Key Sprawl

Many teams struggle with per-team key sprawl where API keys are scattered across services without shared caps. The Respan Gateway allows you to issue unique API keys per environment or team, applying centralized warn/block policies to keep usage under control.

Production-Grade Reliability

Directly calling providers often leads to user-facing downtime when an error occurs. With the Respan Gateway, the "hot path" is protected by automatic failover lists, ensuring that an error from one provider immediately triggers a retry with a fallback model.

Optimized Latency with Caching

For high-traffic applications with frequent repeat prompts, the Respan Gateway cache significantly cuts latency. By configuring cache_ttl and is_cached_by_model, developers can fine-tune how long responses are stored and ensure model-specific accuracy.

How to Use Respan Gateway

Integrating the Respan Gateway into your existing stack is straightforward and requires minimal code changes. Follow these steps to get started:

1. Get your Respan API key: Sign up on the platform and create your first key on the API keys page.
2. Add provider credentials: Connect your providers (OpenAI, Anthropic, etc.) in the Integrations section or add credits on Billing.
3. Choose your mode: Decide whether to use the OpenAI-style unified base URL or native URLs for providers like Gemini.
4. Send parameters: Tag users and enable features like fallbacks and caching via the extra_body parameter.

Implementation Example (Python)

from openai import OpenAI

client = OpenAI(
    base_url="https://api.respan.ai/api/",
    api_key="YOUR_RESPAN_API_KEY",
)

response = client.chat.completions.create(
    model="gpt-5.4",
    messages=[{"role": "user", "content": "Hello!"}],
    extra_body={
        "customer_identifier": "user_123",
        "metadata": {"feature": "chatbot", "environment": "production"},
        "fallback_models": ["claude-sonnet-4-20250514", "gemini-2.5-flash"],
        "cache_enabled": True,
        "cache_ttl": 600,
        "cache_options": {"cache_by_customer": True},
    },
)
print(response.choices[0].message.content)

Security and Compliance

Respan Gateway is committed to the highest standards of data safety and privacy. It is designed to meet the rigorous requirements of global enterprises:

• ISO 27001: Fully compliant with international information security management standards.
• SOC 2: Meets strict requirements for secure data management across all systems.
• GDPR: Operates under the world's strictest data privacy standards for global compliance.
• HIPAA: Offers compliance for healthcare organizations, including available Business Associate Agreements (BAAs).

FAQ

Q: How many models does Respan Gateway support? A: The gateway supports over 500 models through a unified router or provider passthrough, including all major LLM providers.

Q: What happens if a model hits a rate limit? A: If you have configured fallback_models, the Respan Gateway will automatically try the next model in your fallback list to ensure your application remains operational.

Q: Can I limit how much each team spends on AI? A: Yes, you can set soft warnings and hard caps per API key, with notifications sent via Slack or email when thresholds are reached.

Q: Does Respan Gateway store my request data? A: By default, the gateway logs requests and responses for observability. However, you can use disable_log to record metrics only or omit_log to skip logging on cache hits.

Q: Is the caching system secure for multiple users? A: Absolutely. By enabling cache_by_customer, the Respan Gateway ensures that cached responses are only served back to the specific user who generated them, preventing data leakage between customers.