<p align="right"> <strong>English</strong> | <a href="./.github/README.cn.md">中文</a> | <a href="./.github/README.jp.md">日本語</a> </p>

[!IMPORTANT]
🚀 Gateway 2.0 (Pre-Release) Portkey's core enterprise gateway is merging into open-source with our 2.0 release. You can try the pre-release branch here.
Read more about what's next for Portkey in our Series A announcement.

<div align="center">

🆕 Portkey Models - Open-source LLM pricing for 2,300+ models across 40+ providers. Explore →

AI Gateway

Route to 250+ LLMs with 1 fast & friendly API

<img src="https://cfassets.portkey.ai/sdk.gif" width="550px" alt="Portkey AI Gateway Demo showing LLM routing capabilities" style="margin-left:-35px">

Docs | Enterprise | Hosted Gateway | Changelog | API Reference

<a href="https://us-east-1.console.aws.amazon.com/cloudformation/home?region=us-east-1#/stacks/quickcreate?stackName=portkey-gateway&templateURL=https://portkey-gateway-ec2-quicklaunch.s3.us-east-1.amazonaws.com/portkey-gateway-ec2-quicklaunch.template.yaml"><img src="https://img.shields.io/badge/Deploy_to_EC2-232F3E?style=for-the-badge&logo=amazonwebservices&logoColor=white" alt="Deploy to AWS EC2" width="105"/></a>

</div> <br/>

The AI Gateway is designed for fast, reliable & secure routing to 1600+ language, vision, audio, and image models. It is a lightweight, open-source, and enterprise-ready solution that allows you to integrate with any language model in under 2 minutes.

• Blazing fast (<1ms latency) with a tiny footprint (122kb)
• Battle tested, with over 10B tokens processed everyday
• Enterprise-ready with enhanced security, scale, and custom deployments

<br>

What can you do with the AI Gateway?

• Integrate with any LLM in under 2 minutes - Quickstart
• Prevent downtimes through automatic retries and fallbacks
• Scale AI apps with load balancing and conditional routing
• Protect your AI deployments with guardrails
• Go beyond text with multi-modal capabilities
• Explore agentic workflow integrations
• Manage MCP servers with enterprise auth & observability using MCP Gateway

<br><br>

[!TIP]
Starring this repo helps more developers discover the AI Gateway 🙏🏻

<br> <br>

Quickstart (2 mins)

1. Setup your AI Gateway

# Run the gateway locally (needs Node.js and npm)
npx @portkey-ai/gateway

The Gateway is running on http://localhost:8787/v1

The Gateway Console is running on http://localhost:8787/public/

^{Deployment guides: &nbsp; <a href="https://portkey.wiki/gh-18"><img height="12" width="12" src="https://cfassets.portkey.ai/logo/dew-color.svg" /> Portkey Cloud (Recommended)</a> &nbsp; <a href="./docs/installation-deployments.md#docker"><img height="12" width="12" src="https://cdn.simpleicons.org/docker/3776AB" /> Docker</a> &nbsp; <a href="./docs/installation-deployments.md#nodejs-server"><img height="12" width="12" src="https://cdn.simpleicons.org/node.js/3776AB" /> Node.js</a> &nbsp; <a href="./docs/installation-deployments.md#cloudflare-workers"><img height="12" width="12" src="https://cdn.simpleicons.org/cloudflare/3776AB" /> Cloudflare</a> &nbsp; <a href="./docs/installation-deployments.md#replit"><img height="12" width="12" src="https://cdn.simpleicons.org/replit/3776AB" /> Replit</a> &nbsp; <a href="./docs/installation-deployments.md"> Others...</a>}

2. Make your first request

<!-- <details open> <summary>Python Example</summary> -->

# pip install -qU portkey-ai

from portkey_ai import Portkey

# OpenAI compatible client
client = Portkey(
    provider="openai", # or 'anthropic', 'bedrock', 'groq', etc
    Authorization="sk-***" # the provider API key
)

# Make a request through your AI Gateway
client.chat.completions.create(
    messages=[{"role": "user", "content": "What's the weather like?"}],
    model="gpt-4o-mini"
)

^{Supported Libraries:
<img height="12" width="12" src="https://cdn.simpleicons.org/javascript/3776AB" /> JS
<img height="12" width="12" src="https://cdn.simpleicons.org/python/3776AB" /> Python
<img height="12" width="12" src="https://cdn.simpleicons.org/gnubash/3776AB" /> REST
<img height="12" width="12" src="https://cdn.simpleicons.org/openai/3776AB" /> OpenAI SDKs
<img height="12" width="12" src="https://cdn.simpleicons.org/langchain/3776AB" /> Langchain
LlamaIndex
Autogen
CrewAI
More..}

On the Gateway Console (http://localhost:8787/public/) you can see all of your local logs in one place.

<img src="https://github.com/user-attachments/assets/362bc916-0fc9-43f1-a39e-4bd71aac4a3a" width="400" />

3. Routing & Guardrails

Configs in the LLM gateway allow you to create routing rules, add reliability and setup guardrails.

config = {
  "retry": {"attempts": 5},

  "output_guardrails": [{
    "default.contains": {"operator": "none", "words": ["Apple"]},
    "deny": True
  }]
}

# Attach the config to the client
client = client.with_options(config=config)

client.chat.completions.create(
    model="gpt-4o-mini",
    messages=[{"role": "user", "content": "Reply randomly with Apple or Bat"}]
)

# This would always response with "Bat" as the guardrail denies all replies containing "Apple". The retry config would retry 5 times before giving up.

<div align="center"> <img src="https://portkey.ai/blog/content/images/size/w1600/2024/11/image-15.png" width=600 title="Request flow through Portkey's AI gateway with retries and guardrails" alt="Request flow through Portkey's AI gateway with retries and guardrails"/> </div>

You can do a lot more stuff with configs in your AI gateway. Jump to examples →

<br/>

Enterprise Version (Private deployments)

<sup>

<img height="12" width="12" src="https://cfassets.portkey.ai/amazon-logo.svg" /> AWS
<img height="12" width="12" src="https://cfassets.portkey.ai/azure-logo.svg" /> Azure
<img height="12" width="12" src="https://cdn.simpleicons.org/googlecloud/3776AB" /> GCP
<img height="12" width="12" src="https://cdn.simpleicons.org/redhatopenshift/3776AB" /> OpenShift
<img height="12" width="12" src="https://cdn.simpleicons.org/kubernetes/3776AB" /> Kubernetes

</sup>

The LLM Gateway's enterprise version offers advanced capabilities for org management, governance, security and more out of the box. View Feature Comparison →

The enterprise deployment architecture for supported platforms is available here - Enterprise Private Cloud Deployments

<a href="https://portkey.sh/demo-13"><img src="https://portkey.ai/blog/content/images/2024/08/Get-API-Key--5-.png" height=50 alt="Book an enterprise AI gateway demo" /></a><br/>

<br>

MCP Gateway

MCP Gateway provides a centralized control plane for managing MCP (Model Context Protocol) servers across your organization.

• Authentication — Single auth layer at the gateway. Users authenticate once; your MCP servers receive verified requests
• Access Control — Control which teams and users can access which servers and tools. Revoke access instantly
• Observability — Every tool call logged with full context: who called what, parameters, response, latency
• Identity Forwarding — Forward user identity (email, team, roles) to MCP servers automatically

Works with Claude Desktop, Cursor, VS Code, and any MCP-compatible client. Get started →

<br>

Core Features

Reliable Routing

• <a href="https://portkey.wiki/gh-37">Fallbacks</a>: Fallback to another provider or model on failed requests using the LLM gateway. You can specify the errors on which to trigger the fallback. Improves reliability of your application.
• <a href="https://portkey.wiki/gh-38">Automatic Retries</a>: Automatically retry failed requests up to 5 times. An exponential backoff strategy spaces out retry attempts to prevent network overload.
• <a href="https://portkey.wiki/gh-39">Load Balancing</a>: Distribute LLM requests across multiple API keys or AI providers with weights to ensure high availability and optimal performance.
• <a href="https://portkey.wiki/gh-40">Request Timeouts</a>: Manage unruly LLMs & latencies by setting up granular request timeouts, allowing automatic termination of requests that exceed a specified duration.
• <a href="https://portkey.wiki/gh-41">Multi-modal LLM Gateway</a>: Call vision, audio (text-to-speech & speech-to-text), and image generation models from multiple providers — all using the familiar OpenAI signature
• <a href="https://portkey.wiki/gh-42">Realtime APIs</a>: Call realtime APIs launched by OpenAI through the integrate websockets server.

Security & Accuracy

• <a href="https://portkey.wiki/gh-88">Guardrails</a>: Verify your LLM inputs and outputs to adhere to your specified checks. Choose from the 40+ pre-built guardrails to ensure compliance with security and accuracy standards. You can <a href="https://portkey.wiki/gh-43">bring your own guardrails</a> or choose from our <a href="https://portkey.wiki/gh-44">many partners</a>.
• Secure Key Management: Use your own keys or generate virtual keys on the fly.
• Role-based access control: Granular access control for your users, workspaces and API keys.
• <a href="https://portkey.wiki/gh-47">Compliance & Data Privacy</a>: The AI gateway is SOC2, HIPAA, GDPR, and CCPA compliant.

Cost Management

• Smart caching: Cache responses from LLMs to reduce costs and improve latency. Supports simple and semantic* caching.
• Usage analytics: Monitor and analyze your AI and LLM usage, including request volume, latency, costs and error rates.
• Provider optimization*: Automatically switch to the most cost-effective provider based on usage patterns and pricing models.

Collaboration & Workflows

• <a href="https://portkey.ai/docs/integrations/agents">Agents Support</a>: Seamlessly integrate with popular agent frameworks to build complex AI applications. The gateway seamlessly integrates with Autogen, CrewAI, LangChain, LlamaIndex, Phidata, Control Flow, and even Custom Agents.
• Prompt Template Management*: Create, manage and version your prompt templates collaboratively through a universal prompt playground.

<br/><br/>

^{*&nbsp;Available in hosted and enterprise versions} <br>

Portkey Models

Open-source LLM pricing database for 40+ providers - used by the Gateway for cost tracking.

GitHub | Model Explorer

<br>

Cookbooks

☄️ Trending

• Use models from Nvidia NIM with AI Gateway
• Monitor CrewAI Agents with Portkey!
• Comparing Top 10 LMSYS Models with AI Gateway.

🚨 Latest

• Create Synthetic Datasets using Nemotron
• Use the LLM Gateway with Vercel's AI SDK
• Monitor Llama Agents with Portkey's LLM Gateway

View all cookbooks →
<br/><br/>

Supported Providers

Explore Gateway integrations with 45+ providers and 8+ agent frameworks.

	Provider	Support	Stream
<img src="docs/images/openai.png" width=35 />	OpenAI	✅	✅
<img src="docs/images/azure.png" width=35>	Azure OpenAI	✅	✅
<img src="docs/images/anyscale.png" width=35>	Anyscale	✅	✅
<img src="https://upload.wikimedia.org/wikipedia/commons/2/2d/Google-favicon-2015.png" width=35>	Google Gemini	✅	✅
<img src="docs/images/anthropic.png" width=35>	Anthropic	✅	✅
<img src="docs/images/cohere.png" width=35>	Cohere	✅	✅
<img src="https://assets-global.website-files.com/64f6f2c0e3f4c5a91c1e823a/654693d569494912cfc0c0d4_favicon.svg" width=35>	Together AI	✅	✅
<img src="https://www.perplexity.ai/favicon.svg" width=35>	Perplexity	✅	✅
<img src="https://docs.mistral.ai/img/favicon.ico" width=35>	Mistral	✅	✅
<img src="https://docs.nomic.ai/img/nomic-logo.png" width=35>	Nomic	✅	✅
<img src="https://files.readme.io/d38a23e-small-studio-favicon.png" width=35>	AI21	✅	✅
<img src="https://platform.stability.ai/small-logo-purple.svg" width=35>	Stability AI	✅	✅
<img src="https://deepinfra.com/_next/static/media/logo.4a03fd3d.svg" width=35>	DeepInfra	✅	✅
<img src="https://ollama.com/public/ollama.png" width=35>	Ollama	✅	✅
<img src="https://novita.ai/favicon.ico" width=35>	Novita AI	✅	✅

View the complete list of 200+ supported models here

<br>

<br>

Agents

Gateway seamlessly integrates with popular agent frameworks. Read the documentation here.

Framework	Call 200+ LLMs	Advanced Routing	Caching	Logging & Tracing*	Observability*	Prompt Management*
Autogen	✅	✅	✅	✅	✅	✅
CrewAI	✅	✅	✅	✅	✅	✅
LangChain	✅	✅	✅	✅	✅	✅
Phidata	✅	✅	✅	✅	✅	✅
Llama Index	✅	✅	✅	✅	✅	✅
Control Flow	✅	✅	✅	✅	✅	✅
Build Your Own Agents	✅	✅	✅	✅	✅	✅
<img src="https://io.net/favicon.ico" width=35>	IO Intelligence	✅	✅

<br>

*Available on the hosted app. For detailed documentation click here.

Gateway Enterprise Version

Make your AI app more <ins>reliable</ins> and <ins>forward compatible</ins>, while ensuring complete <ins>data security</ins> and <ins>privacy</ins>.

✅ Secure Key Management - for role-based access control and tracking <br>
✅ Simple & Semantic Caching - to serve repeat queries faster & save costs <br>
✅ Access Control & Inbound Rules - to control which IPs and Geos can connect to your deployments <br>
✅ PII Redaction - to automatically remove sensitive data from your requests to prevent indavertent exposure <br>
✅ SOC2, ISO, HIPAA, GDPR Compliances - for best security practices <br>
✅ Professional Support - along with feature prioritization <br>

Schedule a call to discuss enterprise deployments

<br>

Contributing

The easiest way to contribute is to pick an issue with the good first issue tag 💪. Read the contribution guidelines here.

Bug Report? File here | Feature Request? File here

Getting Started with the Community

Join our weekly AI Engineering Hours every Friday (8 AM PT) to:

• Meet other contributors and community members
• Learn advanced Gateway features and implementation patterns
• Share your experiences and get help
• Stay updated with the latest development priorities

Join the next session → | Meeting notes

<br>

Community

Join our growing community around the world, for help, ideas, and discussions on AI.

• View our official Blog
• Chat with us on Discord
• Follow us on Twitter
• Connect with us on LinkedIn
• Read the documentation in Japanese
• Visit us on YouTube
• Join our Dev community

<!-- - Questions tagged #portkey on [Stack Overflow](https://stackoverflow.com/questions/tagged/portkey) -->