frugal

# Frugal <p align="center"><img src="assets/banner.png" alt="Frugal — run AI agents cheap, local, and verified" width="100%"></p> **🇬🇧 English · [🇷🇺 Русский](README.ru.md)** [](https://github.com/SRKRZ23/frugal/actions/workflows/ci.yml) [](LICENSE) [](pyproject.toml)    -00b25c.svg)  [](https://github.com/SRKRZ23/frugal/stargazers) [](https://github.com/SRKRZ23/frugal/commits/main) **Run AI agents cheap, local, and verified.** 🌐 **[Live site](https://frugal-cost-router.netlify.app)** · 🎞 **[Interactive deck](https://frugal-cost-router.netlify.app/deck.html)** Frugal is a drop-in ops layer for LLM apps and agents. It makes any workload: - 💸 **cheaper** — meter every call, cascade-route cheap→frontier, enforce a hard budget - 🏠 **local** — route private/simple work to on-device models (Ollama / vLLM / AMD ROCm) - ✅ **verified** — offline semantic asserts, drift detection, and RAG checks for CI - 🔎 **self-aware** — an MCP server that lets the agent read its own `$/token` spend One package, nine modules, **zero API keys required** — everything runs offline out of the box on a deterministic mock provider, so you can try it, test it, and demo it immediately. ```bash pip install -e . frugal demo # end-to-end showcase, fully offline ``` ## Why Agentic AI sends the bill soaring — every token has a price, and most agents burn frontier tokens on trivial work, leak private prompts to the cloud, and fail *silently* at the semantic level. Frugal is the thin layer that fixes all four in one place, built around a single shared cost **Meter**. ## Architecture Nine modules, one shared cost ledger. Everything you do — route, cache, verify, serve — is metered in the same place, so the agent (and you) can always see the `$/token` truth. ```mermaid flowchart TB subgraph savers["spend less"] R["route — cascade cheap→frontier"] C["cache — repeat prompt = $0"] L["local — on-prem, 0 private leaks"] end subgraph verify["prove it's good enough"] E["eval — asserts · drift · LLM judge"] G["rag — faithfulness / citation checks"] end subgraph observe["see and cap the spend"] MC["mcp — agent reads its own $/token"] GW["gateway — OpenAI-compatible budget proxy"] EC["economics — won't-save warning"] end M[("meter — one shared cost ledger + hard budget cap")] R --> M C --> M L --> M E --> M G --> M MC --> M GW --> M EC --> M ``` ### How routing works The cheap or local model answers first. A **confidence check** decides whether that answer is trustworthy; only the uncertain ones escalate to a frontier model. If the price gap is too small to cover the check, Frugal **warns you instead of quietly overcharging**. ```mermaid flowchart LR P["prompt"] --> CH["cheap / local model"] CH --> K{"confidence<br/>check"} K -->|high| DONE["answer — cost saved"] K -->|low| ESC["escalate to frontier"] ESC --> DONE K -.->|"price gap too small?"| WARN["economics warns you"] ``` ## Watch the bill <p align="center"><img src="assets/charts/cost-curve.png" alt="Cumulative cost: frontier-only $10.20 vs Frugal $0.83 over 2,400 requests — 91.9% saved" width="100%"></p> ```text FRUGAL — live cost (gpt-4o-mini cheap → gpt-4o escalate, real prices) requests processed : 2400 frontier-only : $ 10.2000 ██████████████████████████████████ frugal : $ 0.8262 ██ saved 91.9% ($9.3738 kept in your pocket) ``` `python examples/live_cost_demo.py` (animated) or `--cast` to make an [asciinema](https://asciinema.org) recording (`examples/demo.cast`). Real prices; the exact savings depend on your workload and confidence signal — see the [savings breakdown on the live site](https://frugal-cost-router.netlify.app/#savings). <p align="center"><img src="assets/charts/savings.png" alt="Savings by scenario — 88–97% local, 90.6% with free logprob, down to −7% where cascading loses money" width="100%"></p> Frugal doesn't pretend it always wins: on a small price gap (Haiku→Sonnet, ~3×) a re-sampling check can *cost more* than it saves. The tool computes that and warns you — publishing where it **doesn't** work is why you can trust where it does. ## Verify it yourself in 10 seconds Don't trust the numbers — reproduce them. No API keys needed: ```bash pip install -e . && frugal demo # end-to-end, offline, ~2s python benchmarks/run_all.py # the offline benchmark table python benchmarks/stress_test.py # 8 dims: thread-safety, ReDoS, fuzz, memory python benchmarks/stress_deep.py # 6 adversarial dims: hostile inputs, races (diamond) python benchmarks/cost_model.py # real-price savings math (reproduce the numbers) ``` Have models on a cluster/Ollama? Re-run the exact model comparison on **your** models and dispute it: `FRUGAL_MODELS=... python benchmarks/bench_models.py`. Every claim here is one command away from being checked. (See [WEAKNESSES.md](WEAKNESSES.md) for what the numbers do and don't prove.) ## Modules | Module | What it does | Import | |---|---|---| | `frugal.meter` | per-call cost/token/latency accounting + budget guard | `from frugal import Meter` | | `frugal.route` | cascade routing: cheap model first, escalate only when needed | `from frugal import cascade` | | `frugal.local` | route local↔cloud by cost/privacy/complexity; AMD quickstart | `from frugal.local import LocalRouter` | | `frugal.eval` | `assert_semantic` / `assert_no_hallucination` / `assert_tone` + drift | `from frugal.eval import assert_semantic` | | `frugal.rag` | `ragcheck`: retrieval hit-rate, faithfulness, citation coverage | `from frugal.rag import ragcheck` | | `frugal.mcp` | MCP server exposing live cost telemetry + PII/injection guard | `from frugal.mcp import FrugalMCP` | | `frugal.gateway` | OpenAI-compatible proxy: meter + budget + route (`[gateway]`) | `from frugal.gateway import create_app` | | `frugal.cache` | response cache — the second cost lever; a hit costs $0 | `from frugal import ResponseCache` | | `frugal.economics` | warns when a cheap/frontier pair can't cover the routing overhead | `from frugal.economics import routing_savings` | ## Quickstart ```python from frugal import Meter, MockProvider, cascade from frugal.eval import assert_semantic from frugal.mcp import FrugalMCP provider = MockProvider() # swap for get_openai(...) / get_ollama(...) meter = Meter(budget_usd=0.50) r = cascade("say hi", provider, meter) # -> stays on the cheap model r = cascade("prove this design, step by step", provider, meter) # -> escalates assert_semantic("Paris is the capital", "The capital is Paris", threshold=0.4) print(FrugalMCP(meter).call("get_cost_summary")) ``` ## CLI ```bash frugal demo # offline end-to-end showcase frugal diagnose your_log.jsonl # PROJECT savings on YOUR prompt log — offline, no model called, nothing leaves the box frugal diagnose your_log.jsonl --live --backend ollama --host http://localhost:11434 \ --cheap qwen2.5:7b --frontier phi4:14b # MEASURE it: really route each prompt, observe cost+latency+escalation frugal route "explain this bug" # cascade-route one prompt, print cost frugal rag check examples/rag_example.json frugal mcp # list the MCP cost-telemetry tools frugal gateway --budget 5.00 # OpenAI-compatible budget gateway (needs frugal[gateway]) ``` ## Real providers ```python from frugal.providers import get_openai, get_ollama cloud = get_openai(base_url="https://api.openai.com/v1") # or any OpenAI-compatible endpoint local = get_ollama(model="llama3") # http://localhost:11434 ``` Serve a local model on **AMD ROCm** (MI300X / Radeon) in one command: ```bash MODEL=meta-llama/Llama-3.1-8B-Instruct src/frugal/local/amd_quickstart.sh # -> OpenAI-compatible endpoint at http://localhost:8000/v1 ``` ## Tests ```bash pip install -e '.[dev]' pytest -q ``` ## Benchmarks (measured) Every number is produced by running the code on a labelled synthetic workload — reproduce with `python benchmarks/run_all.py` (writes [`benchmarks/RESULTS.md`](benchmarks/RESULTS.md)). | What | Result | Notes | |---|---|---| | **Cost saved by routing** — *baseline named, no strawman* | **~70–79%** vs a **frontier-default** baseline (gpt-4o-mini→gpt-4o), up to **~97%** with a local $0 tier, and **negative (−3 to −12%)** on a small price gap (Haiku→Sonnet) | vs a **tuned static-cheap** baseline, Frugal costs **more** — its value there is quality-recovery on the escalated ~17% + governance + a measured $/token, **not** a lower bill. Real 2026 prices; run `benchmarks/cost_model.py`. Full method + what's *not* proven: **[METHODOLOGY.md](METHODOLOGY.md)** | | **Private-prompt leaks** to cloud | **0 / 6** | privacy invariant: `private`-tagged prompts always stay local | | Local share of a mixed workload | **96.2%** kept off the paid tier | complexity heuristic favours the cheap tier (tunable) | | Hallucination-flag precision / recall | **1.0 / 1.0** on the labelled set | heuristic gate; small set | | PII + injection guard | **F1 1.0** | 10-sample labelled set | | RAG gate accuracy vs label | **1.0** (5 ex) | aggregate hit/faith 0.6 reflects the 2 planted bad cases | | Metering overhead | **~0.005 ms/call** (~185k calls/s; varies with load) | negligible next to any network call | > **Honest scope:** these prove the **cost arithmetic, routing/escalation logic, > privacy safety, guard accuracy, and overhead** — exactly, offline. They do **not** > prove real-model *answer quality* (the mock's text is synthetic). For quality > retention on real models, run [`benchmarks/bench_cluster.py`](benchmarks/bench_cluster.py) > against your cluster. ### Real-model results (measured on a rented CPU node) Run on a **rented CPU node** via `benchmarks/bench_cluster.py`, using that node's own Ollama models — **`qwen2.5-coder:3b` (cheap) + `qwen2.5:7b` (strong)**, CPU inference, self-consistency confidence, 6 prompts. Single-run, real numbers (model sampling is nondeterministic): | metric | value | |---|---| | cheap-tier latency p50 / p95 | **1.56s / 13.1s** | | strong-tier latency p50 / p95 | **3.70s / 69.1s** (7B on CPU — the tail is why you don't send easy work here) | | cheap≈strong agreement | **4 / 6 (67%)** — with a capable cheap model, most answers already match the strong one | | escalations (self-consistency) | **2 / 6** — only the uncertain prompts escalate | | local token cost | **$0** (on-prem); modelled cloud ≈ $55 / 1M requests at $0.90/1M | An earlier smaller-model stand-in run (`qwen2.5:0.5b` + `qwen2.5:3b`) showed the **confidence-signal finding** starkly: **hedging** confidence escalated **0/12** (real models don't self-hedge — useless signal), while **self-consistency** escalated **9/12**. **Finding (honest):** the routing *mechanism* works; the *confidence signal* decides quality. Default hedging-word confidence is fine for demos but inadequate on real models — use `frugal.route.make_self_consistency(...)` (or logprobs / a verifier). A bigger cheap model raises the cheap≈strong rate (17% with 0.5b → 67% with 3b-coder), so more traffic safely stays cheap. This is what the benchmark surfaces instead of hiding. ### Cross-model comparison (rented node, LLM-judged) <p align="center"><img src="assets/charts/models.png" alt="A 3B model kept up with a 14B at 4.7x the speed — rented node, LLM-judged" width="100%"></p> `benchmarks/bench_models.py` on a **rented CPU node** — five real Ollama models, CPU inference, quality graded by an **LLM judge** (`qwen2.5:7b`) against reference `phi4:14b` (5 prompts, `num_predict=220`): | model | p50 s | p95 s | tok/s | judge quality 0-1 | as-good-as 14B ref | |---|---|---|---|---|---| | **qwen2.5-coder:3b** | **12.7** | 27.6 | 6.5 | 0.80 | **5/5** | | qwen2.5:7b | 20.2 | 59.4 | 3.2 | 0.82 | 5/5 | | deepseek-r1:7b | 59.3 | 70.9 | 3.6 | 0.64 | 5/5 | | phi4:14b (ref) | 60.1 | 116.8 | 1.5 | 0.86 | — | | deepseek-r1:14b | 113.8 | 115.2 | 1.8 | 0.80 | 5/5 | **What this proves (real numbers):** 1. **The cheap tier is enough here.** `qwen2.5-coder:3b` is **~4.7× faster** than `phi4:14b` and the judge rated its answers *as good as* the 14B on **5/5** prompts — so routing there first and escalating rarely cuts latency/cost ~80% at ~no quality loss on this workload. 2. **Reasoning models are a bad default for simple work.** `deepseek-r1:7b` scored the *lowest* quality (0.64) while emitting the *most* tokens (verbose `<think>`), and `deepseek-r1:14b` was the slowest (113s) with no quality gain — exactly the "token tsunami" Frugal exists to route around. 3. **Honest caveats:** N=5, relatively easy prompts → retention saturates at 100%; the 7B judge is lenient; CPU-only on the rented node. Reproduce: `FRUGAL_MODELS=... FRUGAL_BENCH_N=... python benchmarks/bench_models.py`. #### Harder prompts break the 100% saturation Re-run on a **hard** set (coding with edge cases, multi-step reasoning, long-context retrieval — `benchmarks/data_hard.py`, `FRUGAL_PROMPTSET=hard`, N=6): | model | p50 s | judge quality 0-1 | as-good-as 14B ref | |---|---|---|---| | qwen2.5-coder:3b | 34.1 | 0.733 | **5/6 (83%)** | | qwen2.5:7b | 89.0 | 0.75 | 5/6 (83%) | | phi4:14b (ref) | 149.8 | 0.75 | — | On hard tasks the cheap tier's retention drops from **100% → 83%**: it genuinely fails ~1 prompt in 6, which is **exactly the traffic escalation should catch**. It's still 4.4× faster and good enough on 83% — so the routing thesis holds *and* now has a measured escalation rate instead of a saturated 100%. ### GPU vs CPU for the cheap tier (rented GPU) Ran `qwen2.5-coder:3b` on a **modest rented GPU** (`benchmarks/bench_gpu/`), CUDA confirmed (partial offload — a 3B model doesn't fully fit a small card): | cheap tier `coder:3b` | p50 | tok/s | |---|---|---| | rented CPU node | 12.7s | 6.5 | | **modest rented GPU** | **1.13s** | 11.5 | **~11× faster on a modest GPU.** That's the routing thesis's punchline: on any GPU the cheap tier is near-instant, so you route almost everything to it and escalate the ~17% of hard prompts a real confidence check flags. (A bigger card would fit the model fully and go faster still — which is exactly the hardware this project is bottlenecked on; see [Backing](#backing--whats-next).) ### Live bake-off vs a plain proxy `benchmarks/compare_litellm_live.py` on a rented node — same models (`qwen2.5-coder:3b` / `qwen2.5:7b`), same 12 prompts. A proxy (LiteLLM-style) sends every prompt to the strong model; Frugal routes by complexity: | metric | proxy (all→strong) | Frugal (routed) | Frugal saves | |---|---|---|---| | strong-model calls | 12 | **6** | half | | total latency | 373.9s | 252.3s | **−32.5%** | | modelled cost (cloud prices) | $0.01568 | $0.00723 | **−53.9%** | Honest caveats: the baseline is a proxy pass-through (the `litellm` package wouldn't finish installing on the CPU-starved node — the behaviour is identical); a cloud-key run needs your keys. Also `benchmarks/bench_broad.py` adds a **judge panel** (multiple models vote) over a categorized set — on a small model gap (3B↔7B) retention saturates at 100%; still LLM-judged, not human. ## Running on a cluster (models off-box) Frugal is the control layer; the GPUs live on your cluster. Point it there with env vars — no code changes — and the same script that runs offline on the mock now drives real Ollama / vLLM endpoints. See [`docs/CLUSTER.md`](docs/CLUSTER.md). ```bash export FRUGAL_LOCAL_BASE_URL=http://cpu-node:11434 FRUGAL_LOCAL_MODEL=llama3 export FRUGAL_CLOUD_BASE_URL=http://gpu-node:8000/v1 FRUGAL_CLOUD_MODEL=Llama-3.1-70B-Instruct python benchmarks/bench_cluster.py # real latency, tokens, $, quality-retention ``` ### Multi-node distributed routing (two physical machines) `benchmarks/bench_multinode.py` — cheap tier on **node A** (a local machine, `qwen2.5:3b`), strong tier on **node B** (a rented node, `qwen2.5:7b`, reached over an SSH tunnel). LocalRouter decides per prompt; 8-prompt balanced mix: | metric | value | |---|---| | routed to node A (local) / node B (rented) | **7 / 1** | | node A latency p50 | **0.53s** | | node B latency (the one hard prompt) | **50.5s** (7B on a busy CPU node) | | private prompts | 2 | | **private leaked across the network** | **0** | **Proves:** routing genuinely spans machines, keeps p50 low by sending only the hard prompt to the heavy tier, and the **privacy invariant holds over a real network hop** — every `private` prompt stayed on node A. (Threshold is tunable via `FRUGAL_COMPLEXITY_THRESHOLD`.) ## Scenarios Runnable stories under [`examples/scenarios/`](examples/scenarios/), each proving one value: `scenario_coding_agent.py` (budget-capped agent), `scenario_private_data.py` (0 leaks), `scenario_ci_eval.py` (catch a silent regression in CI). ## For enterprises Inference is now **~85% of the enterprise AI budget**. Uber burned its whole 2026 AI-coding budget by April; one firm spent **$500M on Claude in a month**; Palantir's CEO calls token pricing "broken." Frugal routes around it. At **$10M/mo** inference spend that's a modeled **$60–90M/yr** saved. → Full roster (Meta · Amazon · Google · xAI · Uber · Palantir · Klarna …), the savings-by-spend table, and the situations it's built for: **[the Enterprise section on the live site](https://frugal-cost-router.netlify.app/#enterprise)** · **[interactive deck](https://frugal-cost-router.netlify.app/deck.html)**. ## Also in this repo - [`rules/AGENTS.md`](rules/AGENTS.md) — 10 copy-paste rules to make any coding agent cost-aware. - [`awesome-onprem-ai.md`](awesome-onprem-ai.md) — curated list of local-first / on-prem AI tooling. ## Notes on honesty Price figures in `frugal/meter/pricing.py` are **approximate public list prices** and user-editable — verify against your provider before relying on the dollar values. The eval asserts are **cheap heuristic gates** for CI, not ground-truth judges; pass a `provider=` to upgrade them to an LLM judge. ## How Frugal differs (it's not "just a LiteLLM wrapper") Proxies like LiteLLM / Portkey / Helicone route and log across providers. Frugal overlaps there but its point is the **decision + verification loop**, not the proxy: | | LiteLLM / gateways | **Frugal** | |---|---|---| | Multi-provider proxy | ✅ | ✅ (gateway) | | **Cost-aware cascade** (cheap→escalate on a real confidence check) | mostly manual | ✅ measured, with self-consistency | | **Offline eval asserts + RAG checks for CI** | ✕ | ✅ | | **MCP server** exposing the agent's own $/token | ✕ | ✅ | | **Local-first** routing by privacy/complexity | partial | ✅ (0 private leaks, tested) | | **Reproducible model-vs-model benchmark** built in | ✕ | ✅ (`bench_models.py`) | | Zero-dep, runs fully offline (mock) | ✕ | ✅ | If all you need is a provider proxy, use LiteLLM. Frugal is for *deciding what to run and proving it was good enough* — cheaply, locally, in CI. ## Roadmap & maintenance Longevity is the #1 thing developers ask about a new repo, so plainly: this is actively maintained, Apache-2.0, and issues/PRs get triaged. Near-term roadmap: - broader, human-graded, multi-workload evals (retire the "small N" caveat) - logprob/verifier confidence (cheaper than self-consistency) - pre-reservation budget mode (zero-overshoot hard cap) - head-to-head bake-off vs LiteLLM/Portkey - more provider adapters + a persistent metrics store Not promising a foundation — promising honest, responsive upkeep and a public roadmap. ## Security The `frugal.mcp.guard` PII/injection checks are a **first-line heuristic**, not a security product — they catch the obvious and miss the clever; don't make them your only control. Report vulnerabilities via a private issue. See [WEAKNESSES.md](WEAKNESSES.md) for the full honest audit. ## Backing & what's next Frugal is built and benchmarked **solo, on rented commodity CPU/GPU nodes** — which is also the ceiling. The numbers above are CPU-bound or run on a small partial-offload GPU; the single biggest unlock is **hardware**. A full-fit GPU (or an AMD MI300X-class accelerator) makes the cheap tier near-instant and lets the benchmarks graduate from "small N, one rented node" to broad, human-graded, multi-workload proof — in the open. **I'm open to the support that makes that happen:** - 🅰️ **Angel investment** — to fund compute, evaluations, and full-time work on the on-prem AI cost-efficiency stack (Frugal + [REPOMIND v3](https://github.com/SRKRZ23/repomind-v3)). - 🎁 **Non-refundable grants** — research / OSS / hardware grants; no equity, no repayment. - 🖥️ **Hardware sponsorship** — GPU / accelerator access (AMD ROCm-first) to run the bigger benchmarks publicly. - 🤝 **Acquisition / acquihire** — the way strong small teams have joined larger ones for their people and IP (the Wang / Suleyman / Shazeer pattern). Frugal and REPOMIND are one on-prem, cost-efficient-inference thesis — I'd bring both. 📬 **Sardor Razikov** — razikovsardor1@gmail.com · [GitHub](https://github.com/SRKRZ23) · [LinkedIn](https://linkedin.com/in/sardor-razikov-569a5327b) · [X](https://x.com/SardorRazi99093) ## License Apache-2.0.