Content
# Local LLM Delegation MCP
An optimized Model Context Protocol (MCP) server designed to dramatically reduce premium token costs by intelligently delegating low-complexity tasks to high-performance local LLMs via **LiteLLM**.
This server allows you to leverage massive local context windows for "surgical" development tasks (refactoring, unit tests, documentation) while keeping your primary LLM focused on complex architecture and reasoning.
## 🚀 Key Features
- **LiteLLM Integration:** Seamlessly connect to Ollama, vLLM, LM Studio, Anthropic, or any OpenAI-compatible provider.
- **Externalized Configuration:** Fine-tune model parameters (temperature, top_p, max_tokens) via `config.yaml`.
- **Customizable Prompts:** Define your own task templates and system messages in `prompts.yaml`.
- **Token Optimization:** Automatically offload simple tasks to local models like Qwen 2.5/3.5 Coder.
- **Usage Tracking:** Monitor your savings with built-in usage logging and statistics.
## 🛠️ Requirements
- **Local LLM Runner** (e.g., Ollama, vLLM, LM Studio)
- **Python 3.13+**
## 📦 Installation
1. **Clone the repository:**
```bash
git clone https://github.com/mrrodriguez/local-llm-delegation-mcp.git
cd local-llm-delegation-mcp
```
2. **Set up environment variables:**
```bash
cp .env.example .env
# Edit .env to match your local setup
```
3. **Choose your installation method:**
**Option A: Global CLI (Recommended for simplicity)**
Install as a global tool. This adds the `local-llm-delegator` command to your PATH.
```bash
pip install .
# Or using pipx (preferred)
pipx install .
```
**Option B: Local Environment (Recommended for development)**
Use a virtual environment and point your MCP client directly to a startup script.
```bash
python -m venv .venv
source .venv/bin/activate
pip install .
```
## ⚙️ Configuration
The server uses a flexible configuration system. Settings are loaded from `config.yaml` and `prompts.yaml` located in the project root, with key overrides available via Environment Variables (or a `.env` file).
### Configuration Hierarchy
1. **Base Defaults:** Hardcoded in `config.py`.
2. **YAML Files:** `config.yaml` (model settings) and `prompts.yaml` (task templates).
3. **Environment Variables:** Overrides specific settings like `LOCAL_MODEL_NAME` and `OPENAI_BASE_URL`.
### 1. Quick Start: Environment Variables
The fastest way to switch models without editing files:
```bash
# In your .env file or shell environment
LOCAL_MODEL_NAME=ollama/llama3:8b # Overrides config.yaml
OPENAI_BASE_URL=http://localhost:11434/v1
```
### 2. Model Tuning (`config.yaml`)
Update the `name` to match your local provider's format (e.g., `ollama/qwen2.5-coder`). The provided template is optimized for high-end hardware (e.g., Apple M-series Max).
```yaml
model:
name: "openai/Qwen3-Coder-30B-A3B-Instruct-MLX-8bit" # Example: OMLX/MLX
temperature: 0.1
max_tokens: 32768
extra_params:
num_ctx: 65536
num_predict: 32768
top_p: 0.80
```
*Note: The `name` must follow the [LiteLLM Provider Format](https://docs.litellm.ai/docs/providers).*
### 3. Custom Tasks (`prompts.yaml`)
Define your own tools by adding entries to `prompts.yaml`. Each top-level key becomes a sub-task for the `query_local_llm_with_context` tool.
## 🔌 Integration
You can add this server to your preferred client using either the global command or a direct path to the project.
### 1. Gemini Code CLI
**If installed globally (Option A):**
```bash
gemini mcp add local-llm-delegator local-llm-delegator --trust
```
**If using a direct script or path (Option B):**
```bash
gemini mcp add local-llm-delegator /path/to/your/start-script.sh --trust
```
### 2. Claude Code
Add to your `~/.claude.json`:
```json
"mcpServers": {
"local-llm-delegator": {
"command": "local-llm-delegator" // If installed globally; otherwise any wrapper script you have to start it
}
}
```
*Note: If using a custom startup script (like for OMLX), set the `command` to the absolute path of your script.*
## 📊 Usage Tracking
The server logs usage to `mcp_usage.jsonl`. View stats via:
- `get_local_llm_usage_stats` (MCP tool)
- `show-stats` (CLI command - coming soon) or `python show_stats.py`
## 📜 License
MIT
