Content
<div align="center">
<img src="art/free-ai-gateaway.png" alt="Free-AI Gateway Architecture" width="100%" />
# ⚡ Free-AI Gateway
**Enterprise-grade capability-routed AI Gateway monorepo aggregating free-tier and local AI APIs into reusable libraries, Model Context Protocol (MCP) servers, and OpenAI-compatible HTTP proxies.**
[](LICENSE)
[](https://www.typescriptlang.org/)
[](https://nodejs.org/)
[](https://fastify.dev/)
[](Dockerfile)
[](LEARN.md)
[](CONTRIBUTING.md)
</div>
> 📚 **New to Free-AI Gateway?** Check out the comprehensive **[Architecture & Developer Guide (LEARN.md)](LEARN.md)** for detailed deep dives, tutorials, and integration patterns.
---
## 📖 Architecture & Monorepo Overview
**`free-ai-gateway`** is organized as an enterprise monorepo separating **pure AI orchestration infrastructure** from **protocol-specific delivery mechanisms** (HTTP Fastify Proxy, MCP Server, CLI, & Web Examples):
```mermaid
flowchart TD
subgraph CoreLayer ["@free-ai-gateway/core (Standalone npm package)"]
Router["CapabilityRouter & Strategy Engine"]
Providers["20 Provider Adapters & Dynamic Registry"]
Resilience["QuotaTracker & CircuitBreaker"]
Observability["EventBus & MetricsTracker"]
Transport["HttpClient with Exponential Backoff"]
end
subgraph Consumers ["Consumer Applications & Examples"]
GatewayApp["apps/gateway (@free-ai-gateway/gateway)<br/>Fastify HTTP OpenAI Proxy & SSE Streamer"]
McpApp["packages/mcp (@free-ai-gateway/mcp)<br/>Model Context Protocol Server"]
SkillsPkg["packages/skills (@free-ai-gateway/skills)<br/>Agentic IDE Skills & CLI"]
CliApp["packages/cli (@free-ai-gateway/cli)<br/>Terminal Assistant & Diagnostics"]
NextApp["examples/nextjs-chat<br/>Next.js 14+ Fullstack Streaming UI"]
Collections["examples/collections<br/>Postman & Bruno API Test Suites"]
end
GatewayApp -->|consumes| CoreLayer
McpApp -->|consumes| CoreLayer
SkillsPkg -->|integrates with| CoreLayer
CliApp -->|consumes| CoreLayer
NextApp -->|proxies to| GatewayApp
Collections -->|tests| GatewayApp
```
### Monorepo Workspaces Matrix
| Package / App | Location | Purpose | Dependencies |
| :--- | :--- | :--- | :--- |
| **`@free-ai-gateway/core`** | `packages/core` | Protocol-neutral capability router, resilience engine, and 20 provider adapters. | `ajv`, `dotenv` (Zero HTTP server) |
| **`@free-ai-gateway/mcp`** | `packages/mcp` | Model Context Protocol server exposing capability tools to AI agents (Claude Desktop, Cursor). | `@free-ai-gateway/core` |
| **`@free-ai-gateway/skills`** | `packages/skills` | Agentic IDE skills (`SKILL.md`) and installer CLI for Antigravity, Claude, Cursor, and Copilot. | Standalone CLI & API |
| **`@free-ai-gateway/cli`** | `packages/cli` | Terminal AI assistant, interactive chat REPL, model catalog, and diagnostics tool. | `@free-ai-gateway/core`, `@free-ai-gateway/skills` |
| **`@free-ai-gateway/gateway`** | `apps/gateway` | High-throughput Fastify HTTP proxy serving OpenAI-compatible endpoints with auto-discovery and SSE. | `@free-ai-gateway/core`, `fastify` |
| **`examples/nextjs-chat`** | `examples/nextjs-chat` | Fullstack Next.js 14+ App Router demo with Vercel AI SDK & zero-dep native SSE streams. | `ai`, `openai`, `next`, `react` |
| **`examples/collections`** | `examples/collections` | Complete Postman v2.1 and Bruno collections covering all 19 API endpoints. | Standard JSON / `.bru` suites |
---
## ✨ Key Capabilities
- 🎯 **Capability-Based Routing**: Request what you need (`model: "auto:tool_calling+structured_output"`), and let the router choose the fastest healthy free provider.
- 📐 **Strategy Pattern Engine**: Pluggable load balancing strategies (`AdaptiveHealthStrategy`, `LowestLatencyStrategy`, or custom `IRoutingStrategy`).
- 🔄 **Autonomous Failover**: Transparently cycles through ranked candidate providers until success upon encountering upstream `429` (Rate Limit) or `5xx` errors.
- 🛡️ **Circuit Breaker**: Detects failing providers and enters exponential cooldown backoff to prevent cascade failures.
- ⏱️ **Sliding-Window Quota Tracking**: In-memory accounting of RPM, TPM, and RPD with proactive limit protection.
- 🔑 **Multi-Key Pool Rotation**: Configure comma-separated API keys (`PROVIDER_KEY=k1,k2,k3`) with round-robin rotation and per-key rate-limit isolation.
- 🔌 **Dynamic Provider Autoloader**: Add new providers by dropping a class extending `BaseProvider` into `packages/core/src/providers/`.
- 📡 **Typed Event Bus & Observability**: Lifecycle events for OpenTelemetry and zero-overhead Prometheus `/metrics` endpoint.
- 🤖 **Model Context Protocol (MCP) Ready**: Use directly in Claude Desktop, Cursor, or agent workflows.
---
## 🧩 Supported Providers Matrix (20 Adapters)
| Provider | Modalities / Capabilities | Authentication | Limit Scope |
| :--- | :--- | :--- | :--- |
| **Google AI Studio** | `text`, `tool_calling`, `vision`, `structured_output`, `embedding`, `tts` | `GOOGLE_API_KEY` | Per Model |
| **Groq** | `text`, `tool_calling`, `structured_output`, `reasoning`, `speech_to_text` | `GROQ_API_KEY` | Account |
| **Ollama (Local)** | `text`, `code`, `reasoning` | None (Local Daemon) | Local Node |
| **SambaNova Cloud** | `text`, `tool_calling`, `reasoning`, `vision` | `SAMBANOVA_API_KEY` | Account |
| **NVIDIA NIM** | `text`, `tool_calling`, `reasoning`, `vision`, `embedding`, `rerank`, `moderation` | `NVIDIA_API_KEY` | Account |
| **Cohere** | `text`, `tool_calling`, `structured_output`, `reasoning`, `embedding`, `rerank` | `COHERE_API_KEY` | Account |
| **OpenRouter** | `text`, `tool_calling`, `vision`, `reasoning`, `embedding`, `tts`, `moderation` | `OPENROUTER_API_KEY` | Account |
| **OpenCode Zen** | `code`, `tool_calling`, `reasoning`, `text` | `OPENCODE_API_KEY` | Account |
| **Bazaarlink.ai** | `text`, `code` | `BAZAARLINK_API_KEY` | Account |
| **aimlapi.com** | `text` | `AIMLAPI_API_KEY` | Account |
| **OVHcloud AI** | `text` | `OVHCLOUD_API_KEY` | Per Model |
| **Voyage AI** | `embedding` | `VOYAGE_API_KEY` | Account |
| **Jina AI** | `embedding`, `rerank` | `JINA_API_KEY` | Account |
| **Hugging Face** | `text`, `tool_calling`, `image_gen` | `HUGGINGFACE_API_KEY` | Shared Pool |
| **Cloudflare Workers AI** | `image_gen`, `embedding` | `CLOUDFLARE_API_TOKEN` | Shared Pool |
| **Google Cloud Platform** | `translation`, `speech_to_text`, `text_to_speech`, `vision` | `GCP_API_KEY` | Account |
| **MyMemory** | `translation` | `MYMEMORY_API_KEY` | Account |
| **Unstructured.io** | `document_processing` | `UNSTRUCTURED_API_KEY` | Account |
| **Exa AI** | `web_search` | `EXA_API_KEY` | Account |
| **Tavily** | `web_search` | `TAVILY_API_KEY` | Account |
---
## 🚀 Quick Start
### 1. Installation
```bash
# Clone the repository
git clone https://github.com/zaber-dev/free-ai-gateway.git
cd free-ai-gateway
# Install dependencies across all monorepo workspaces
npm install
```
### 2. Configure Environment
Copy `.env.example` to `.env` and provide keys for the providers you wish to enable:
```bash
cp .env.example .env
```
```env
PORT=3000
GROQ_API_KEY=gsk_...
GOOGLE_API_KEY=AIza...
NVIDIA_API_KEY=nvapi-...
COHERE_API_KEY=...
```
### 3. Build & Run
```bash
# Compile all workspace packages
npm run build
# Run all 55 automated tests across all packages
npm test
# Start the Fastify HTTP Gateway (Dev mode)
npm run dev
# Start the Gateway in Production
npm start
```
---
## 💻 Usage Modalities
### Option A: HTTP Gateway (OpenAI Compatible)
Call the local proxy with any OpenAI SDK or `curl`:
```bash
curl http://localhost:3000/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{
"model": "auto:tool_calling+structured_output",
"messages": [
{ "role": "user", "content": "Extract name and age from: Alice is 30 years old." }
]
}'
```
```typescript
import OpenAI from "openai";
const client = new OpenAI({
baseURL: "http://localhost:3000/v1",
apiKey: "not-needed",
});
const completion = await client.chat.completions.create({
model: "auto:reasoning",
messages: [{ role: "user", content: "Solve: How many r's in strawberry?" }],
});
console.log(completion.choices[0].message.content);
```
For a runnable Python version using the official OpenAI SDK and LangChain, see [examples/python-langchain](examples/python-langchain/README.md).
---
### Option B: Embedding `@free-ai-gateway/core` as a TypeScript Library
Embed the capability router directly into your application without launching an HTTP server:
```typescript
import {
CapabilityRouter,
Registry,
QuotaTracker,
CircuitBreaker,
EventBus,
LowestLatencyStrategy,
} from "@free-ai-gateway/core";
const registry = new Registry();
const quota = new QuotaTracker();
const breaker = new CircuitBreaker();
const eventBus = new EventBus();
// Listen to lifecycle telemetry
eventBus.on("request:fallback", (evt) => {
console.warn(`[Fallback] Failed on ${evt.attemptedProvider}: ${evt.error}`);
});
const router = new CapabilityRouter(
registry,
quota,
breaker,
undefined,
eventBus,
new LowestLatencyStrategy()
);
const response = await router.route({
capabilities: ["text", "tool_calling"],
payload: {
messages: [{ role: "user", content: "Hello AI!" }],
},
});
console.log("Served by:", response.servedBy);
console.log("Data:", response.data);
```
---
### Option C: Model Context Protocol (MCP) Server
Connect Free-AI Gateway to Claude Desktop or Cursor:
```json
{
"mcpServers": {
"free-ai-gateway": {
"command": "node",
"args": ["/path/to/free-ai-gateway/packages/mcp/dist/index.js"],
"env": {
"GROQ_API_KEY": "gsk_...",
"GOOGLE_API_KEY": "AIza..."
}
}
}
}
```
**Exposed MCP Tools:**
* `freeai_generate`: Generate text, reasoning, or code with automatic failover.
* `freeai_search`: Web search queries via Exa / Tavily.
* `freeai_embed`: Generate vector embeddings via Voyage, Jina, Gemini.
* `freeai_rerank`: Rerank documents for retrieval augmented generation (RAG).
* `freeai_analyze_image`: Multimodal vision analysis.
---
### Option D: Agentic IDE Skills (`@free-ai-gateway/skills`)
Install Free-AI Gateway agent skills directly into your IDE or autonomous coding assistant:
```bash
# Install to Google Antigravity (.agents/skills)
npx @free-ai-gateway/skills install --target=antigravity
# Install to Cursor (.cursor/skills)
npx @free-ai-gateway/skills install --target=cursor
# Install to Claude Code (.claude/skills)
npx @free-ai-gateway/skills install --target=claude
# Install to all supported AI assistants
npx @free-ai-gateway/skills install --target=all
```
---
### Option E: Terminal CLI Tool (`@free-ai-gateway/cli`)
Use Free-AI directly from your terminal or command-line scripts:
```bash
# One-off prompt execution with auto-routing
npx @free-ai-gateway/cli "Explain MapReduce in simple terms"
# Interactive chat REPL in terminal
npx @free-ai-gateway/cli chat --capability=reasoning
# Check model catalog with formatting & capability filter
npx @free-ai-gateway/cli models --format=markdown --capability=code
# Run system diagnostics
npx @free-ai-gateway/cli doctor
```
---
## 🏛️ Monorepo Structure
```
free-ai-gateway/
├── packages/
│ ├── core/ # @free-ai-gateway/core
│ │ ├── AGENTS.md # Agentic guidelines for @free-ai-gateway/core
│ │ ├── src/
│ │ │ ├── capabilities/ # Capability definitions & parsing
│ │ │ ├── config/ # providers.json, schema, config sources
│ │ │ ├── errors/ # ProviderError, NoProviderAvailableError
│ │ │ ├── observability/ # EventBus, MetricsTracker
│ │ │ ├── providers/ # 20 Provider Adapters + Registry + Loader
│ │ │ ├── resilience/ # QuotaTracker, CircuitBreaker
│ │ │ ├── router/ # CapabilityRouter & Strategy Pattern
│ │ │ ├── transport/ # HttpClient with exponential backoff
│ │ │ ├── types/ # Unified contracts & response schemas
│ │ │ └── index.ts # Public Core API
│ │ ├── tests/ # 28 Core unit tests
│ │ └── package.json
│ │
│ ├── mcp/ # @free-ai-gateway/mcp
│ │ ├── AGENTS.md # Agentic guidelines for @free-ai-gateway/mcp
│ │ ├── src/
│ │ │ ├── tools/ # generate, search, embed, rerank, analyze-image
│ │ │ ├── resources/ # capabilities, models catalog
│ │ │ ├── server.ts # FreeAiMcpServer handler
│ │ │ └── index.ts
│ │ ├── tests/ # 3 MCP server tests
│ │ └── package.json
│ │
│ ├── skills/ # @free-ai-gateway/skills
│ │ ├── AGENTS.md # Agentic guidelines for @free-ai-gateway/skills
│ │ ├── src/
│ │ │ ├── skills/ # Built-in skills (free-ai-gateway, scaffolding, mcp)
│ │ │ ├── installer.ts # Multi-target installer
│ │ │ ├── cli.ts # CLI executable (free-ai-skills)
│ │ │ └── index.ts
│ │ ├── tests/ # 4 Skills tests
│ │ └── package.json
│ │
│ └── cli/ # @free-ai-gateway/cli
│ ├── AGENTS.md # Agentic guidelines for @free-ai-gateway/cli
│ ├── src/
│ │ ├── commands/ # prompt, chat, models, doctor, skills
│ │ ├── cli.ts # Argument parsing & dispatcher
│ │ ├── bin.ts # CLI executable (free-ai, freeai)
│ │ └── index.ts
│ ├── tests/ # 8 CLI tests
│ └── package.json
│
├── apps/
│ └── gateway/ # @free-ai-gateway/gateway (HTTP App)
│ ├── AGENTS.md # Agentic guidelines for @free-ai-gateway/gateway
│ ├── src/
│ │ ├── adapters/ # OpenAI chat response normalizer & SSE chunker
│ │ ├── api/
│ │ │ ├── routes/ # Fastify route modules & RouteLoader
│ │ │ └── server.ts # Server factory, timing hooks, 404 handler
│ │ ├── jobs/ # Background JobScheduler & reverify worker
│ │ └── index.ts
│ ├── tests/ # 7 Gateway HTTP tests
│ ├── Dockerfile # Monorepo container builder
│ └── package.json
│
├── examples/
│ ├── nextjs-chat/ # Next.js 14+ App Router & Vercel AI SDK example
│ └── collections/ # Ready-to-import Postman & Bruno API suites
│
├── tests/
│ └── e2e/ # 5 Cross-package E2E integration tests
│
├── AGENTS.md # Monorepo Root Agentic Guidelines
├── CLAUDE.md # Claude Code Instructions
├── .agents/ # Workspace Skills Directory
├── .github/workflows/ci.yml # Matrix CI workflow
├── docker-compose.yml
├── package.json # Root workspace definition
├── tsconfig.base.json # Shared TypeScript compiler settings
└── README.md
```
---
## 🤝 Community & Governance
- 📖 **[Architecture Blueprint](ARCHITECTURE.md)**: Deep dive into the internal system design and data flow.
- 🎓 **[Developer & Learning Guide](LEARN.md)**: Tutorials, programmatic usage, and SDK patterns.
- 🗺️ **[Product Roadmap](ROADMAP.md)**: Planned milestones, distributed state, and upcoming features.
- 💬 **[Support Guide](SUPPORT.md)**: Troubleshooting, community discussions, and help channels.
- 🏛️ **[Project Governance](GOVERNANCE.md)**: Decision-making process, maintainer roles, and release policies.
- ✍️ **[Contributing Guide](CONTRIBUTING.md)**: Step-by-step instructions for adding new provider adapters.
- 🔒 **[Security Policy](SECURITY.md)**: Vulnerability disclosure guidelines.
- 📜 **[Code of Conduct](CODE_OF_CONDUCT.md)**: Community standards and expectations.
---
## 👤 Author
Created and maintained with ❤️ by **[Md. Mahedi Zaman Zaber](https://github.com/zaber-dev)**.
---
## 📄 License
This project is open source and available under the [MIT License](LICENSE).
Connection Info
You Might Also Like
No recommendations available