Content
<p align="center">
<a href="https://github.com/Mike-E-Log/ai-engineer-best-practices/actions/workflows/ci.yml"><img alt="CI" src="https://img.shields.io/github/actions/workflow/status/Mike-E-Log/ai-engineer-best-practices/ci.yml?style=flat-square&label=CI&labelColor=0f172a"/></a>
<img alt="MCP%20tools" src="https://img.shields.io/badge/MCP%20tools-8-7c3aed?style=flat-square&labelColor=0f172a"/>
<img alt="eval%20recipes" src="https://img.shields.io/badge/eval%20recipes-5-0ea5e9?style=flat-square&labelColor=0f172a"/>
<img alt="judges" src="https://img.shields.io/badge/judges-claude%20%2B%20gpt%20%2B%20gemini-f59e0b?style=flat-square&labelColor=0f172a"/>
<img alt="license" src="https://img.shields.io/badge/license-MIT-22c55e?style=flat-square&labelColor=0f172a"/>
</p>
<h2 align="center">ai-engineer-best-practices · handbook + score MCP tool</h2>
<p align="center">
<em>An opinionated, citation-rich reference for working AI engineers — methodology,<br/>patterns, and recipes, with a working <code>score</code> MCP tool that demonstrates a 3-vendor<br/>judge ensemble with agreement metrics and an experimental anti-collusion check.</em>
</p>
### → [Read the handbook](https://mike-e-log.github.io/ai-engineer-best-practices/)
---
## What it is, in one MCP call
`score` grades any AI artifact (prompt, output, eval recipe, anything textual) against a calibrated rubric using a 3-vendor judge ensemble — Claude + GPT + Gemini, scored in parallel, with a dual-gate Kendall-τ ensemble (rank-agreement floor + an experimental anti-collusion ceiling) that flags vendor-correlated false-consensus.
```python
# From any MCP client — Claude Desktop, Cursor, the mcp Python SDK
result = score(
eval_id="kendall-tau-agreement",
artifact="your AI output here",
)
# → per-judge scores, mean, Kendall-τ, PASS / FAIL_MEAN / FAIL_TAU / FAIL_DIM verdict
```
**Example output:**
```json
{
"eval_id": "kendall-tau-agreement",
"verdict": "PASS",
"mean": 8.33,
"kendall_tau": 0.82,
"judges": {
"claude": {"score": 9, "critique": "strong specificity with Kendall-tau and cross-vendor verification"},
"gpt": {"score": 8, "critique": "concrete eval markers; slightly over-relies on punchline phrasing"},
"gemini": {"score": 8, "critique": "balances deep technical signals with a memorable proof-oriented kicker"}
},
"vendor_bias_flag": false,
"anti_collusion_tau_pairs": {"claude_gpt": 0.5, "claude_gemini": 0.82, "gpt_gemini": 0.82}
}
```
`verdict` distinguishes `PASS`, `FAIL_MEAN`, `FAIL_TAU` (rank disagreement), and `FAIL_DIM` (any per-dimension floor breached). `vendor_bias_flag: true` surfaces when any two judges agree above the τ ≤ 0.95 ceiling — a heuristic for human review of vendor-correlated false-consensus, **experimental**, not a validated threshold.
**Cost:** ~$0.15 per call (3 paid judge invocations).
**Auth:** `PEB_MCP_TOKEN` required for the cost-bearing `score` tool.
`score` is the **methodology demo**. The handbook, the patterns, and the 5 calibrated eval recipes it reads from are the reference; `score` is the working proof that the eval discipline is real, not decoration.
## What this isn't
Not another multi-provider LLM wrapper. That layer is solved:
- [LiteLLM](https://github.com/BerriAI/litellm) (~46k★) — call any vendor with one API
- [AISuite](https://github.com/andrewyng/aisuite) (~13k★) — same thesis, lighter
You don't need a sixth wrapper. The wrappers ship the *transport*. They don't carry the *prompts*. And they don't grade what runs through them.
## What it is
The layer **above** the wrappers — the prompts, patterns, anti-patterns, and eval recipes the wrappers carry, with three properties no existing reference combines:
1. **Canonical** — one schema, one source-of-truth file (`patterns.json`), one auto-rendered handbook. CI fails if the handbook drifts from the data.
2. **Agent-pingable** — exposed over an MCP stdio server so any LLM client can call `get_pattern("role-and-rules")` or `score(eval_id, artifact)` mid-conversation.
3. **Eval-disciplined** — every promoted entry and every `score` call clears a 3-vendor judge ensemble with a Kendall-τ agreement gate. Most "best prompts" lists are someone's vibes. This one isn't.
## What's missing in existing repos
These three capabilities exist separately. Their intersection was empty across the top 20+ candidates audited:
| Project | Canonical reference? | MCP-callable? | Eval-graded entries? |
|---|:---:|:---:|:---:|
| [`dair-ai/Prompt-Engineering-Guide`](https://github.com/dair-ai/Prompt-Engineering-Guide) (74k★) | ✅ | ❌ | ❌ |
| [`microsoft/generative-ai-for-beginners`](https://github.com/microsoft/generative-ai-for-beginners) (110k★) | ✅ | ❌ | ❌ |
| [`f/prompts.chat`](https://github.com/f/awesome-chatgpt-prompts) (162k★) | curated | ❌ | ❌ |
| [`anthropics/anthropic-cookbook`](https://github.com/anthropics/anthropic-cookbook) | ✅ recipes | ❌ | ❌ |
| [`gdli6177/mcp-prompt-server`](https://github.com/gdli6177/mcp-prompt-server) | ❌ | ✅ | ❌ |
| **`ai-engineer-best-practices`** | **✅** | **✅** | **✅** |
Existing repos cover the transport layer. This is the reference + grading layer above it.
## MCP tools
After connecting the stdio server (see Install below), an LLM client can call:
| Tool | What it returns |
|---|---|
| `list_patterns(category=...)` | summary of all entries (or one category) — categories: `principles`, `patterns`, `anti_patterns`, `showcase`, `eval_recipes` |
| `get_pattern("<id>")` | full entry by id (resolves any category, including eval recipes) |
| `search("<query>")` | free-text across title / summary / scaffold |
| `showcase()` | top entries from the Showcase slot, ranked by mean eval score |
| `get_schema()` | the JSON Schema `patterns.json` validates against |
| `list_evals(judge_pattern=...)` | list eval recipe summaries, filterable by judge pattern |
| `get_eval_recipe("<id>")` | fetch a full eval recipe by id (with `list_evals` hint on 404) |
| `score(eval_id, artifact)` | **Cost-bearing.** Grade an artifact against an eval recipe using the 3-vendor judge ensemble. Returns per-judge scores, mean, Kendall-τ, and a PASS / FAIL_MEAN / FAIL_TAU / FAIL_DIM verdict. Requires `PEB_MCP_TOKEN`. ~$0.15 per call. |
## How the eval bar works
Three judges, **three different model families** — `claude-opus`, `gpt-frontier`, `gemini-pro` — score each candidate 1–10 on five dimensions: universality, pivotal-ness, reproducibility, outcome legibility, safety. An entry (or any artifact passed through `score`) is rejected unless:
| Gate | Threshold |
|---|---|
| Mean across all dimensions | ≥ 8.0 |
| Lowest single dimension | ≥ 6 |
| **Kendall-τ across the 3 judges** | **≥ 0.5** |
| τ between any two judges | ≤ 0.95 — flags vendor-correlated agreement; `score` returns `vendor_bias_flag: true` for human review (**experimental** heuristic, not a validated threshold) |
The Kendall-τ floor is the most novel piece — it forces agreement on the *relative* ranking of a prompt's strengths and weaknesses, not just on absolute scores. Same-vendor judges produce false agreement on numbers; cross-vendor + rank-correlation catches a specific failure mode where judges agree on scores but disagree on shape. It's a coarse instrument on a 5-dimension ranking; the mean-≥-8.0 gate and the per-dimension floor (≥ 6) carry most of the rejection weight. The anti-collusion ceiling (τ ≤ 0.95) is an opinionated experimental signal, not a settled bar. See [`docs/eval-rubric.md`](docs/eval-rubric.md) for limitations.
Implementation: [`judges.py`](judges.py) (async cross-vendor ensemble) + [`scoring.py`](scoring.py) (Kendall-τ minimum across all judge pairs) + [`tests/eval_judge_prompt.py`](tests/eval_judge_prompt.py) (calibration set that gates any change to the judge prompt template). Full rubric in [`docs/eval-rubric.md`](docs/eval-rubric.md).
The Showcase slot ships empty — the bar held first.
## What's inside
- **10 principles** — foundational rules with failure-mode rationale (e.g. *make the objective explicit*, *iterate with evals not vibes*)
- **10 patterns** — reusable scaffolds with `when_to_use` + edge cases (Role & Rules, Chain-of-Thought, Extractor + Validator Pair, Judge Prompt, …)
- **5 anti-patterns** — production failure modes (kitchen-sink instructions, vague success criteria, untested prompt changes, …)
- **5 eval recipes** — calibrated evaluation methods: `kendall-tau-agreement`, `vendor-bias-detection`, `llm-as-judge-single`, `pairwise-preference`, `prompt-calibration-test`. Each one is what `score` runs against, with practitioner citations (Hamel Husain, Shreya Shankar, LMSYS).
- **Showcase slot** — capped at 6, currently 0/6 (eval bar held first)
- **MCP stdio server, 8 tools** — see table above
Browse the rendered handbook at [`docs/handbook.md`](docs/handbook.md). Interactive view at [`docs/index.html`](docs/index.html).
## 30-second use (MCP)
Add to `claude_desktop_config.json` (or any MCP-compatible client config):
```json
{
"mcpServers": {
"aiebp": {
"command": "peb-server",
"args": []
}
}
}
```
`score` is cost-bearing and requires `PEB_MCP_TOKEN`:
```bash
export PEB_MCP_TOKEN="$(openssl rand -hex 32)"
```
Verified client recipes: [`docs/integrations/claude-desktop.md`](docs/integrations/claude-desktop.md), [`docs/integrations/cursor.md`](docs/integrations/cursor.md), [`docs/integrations/mcp-python-sdk.md`](docs/integrations/mcp-python-sdk.md).
Restart your client. The 8 tools above become callable.
## Install
```bash
git clone https://github.com/Mike-E-Log/ai-engineer-best-practices.git
cd ai-engineer-best-practices
pip install -e .[dev]
pytest # full suite, no API keys required
peb-server # MCP stdio server
```
If `pip install` warns that scripts went to a directory not on PATH (common on Windows), invoke the module directly instead:
```bash
python -m server # equivalent to peb-server
```
For the integration calibration test (live judge ensemble), set `ANTHROPIC_API_KEY`, `OPENAI_API_KEY`, and `GOOGLE_API_KEY` and run `pytest tests/eval_judge_prompt.py`.
## AI Eval
This repo keeps its AI-eval treatment deliberately shallow — enough to ground the eval recipes and the `score` demo. For the deep treatment (practitioner map, Techniques Index), see [ai-eval-atlas](https://github.com/Mike-E-Log/ai-eval-atlas). Part of an AI-evaluation engineering portfolio — overview at [mikeilog.com](https://mikeilog.com).
## Docs
- [`docs/SPEC.md`](docs/SPEC.md) — design rationale, data model, review decisions
- [`docs/handbook.md`](docs/handbook.md) — rendered from `patterns.json` (CI fails on drift)
- [`docs/eval-rubric.md`](docs/eval-rubric.md) — scoring dimensions + thresholds
- [`docs/index.html`](docs/index.html) — interactive showcase view (rendered at the Pages URL above)
- [`CONTRIBUTING.md`](CONTRIBUTING.md) — how to add an entry (including eval recipes — see [eval recipe template](docs/templates/eval-recipe.template.jsonc))
- [`CHANGELOG.md`](CHANGELOG.md) — release notes
## License
MIT. See [`LICENSE`](LICENSE).
---
**AI-evaluation engineering portfolio** — five repos, one discipline:
- [**ai-eval-toolkit**](https://github.com/Mike-E-Log/ai-eval-toolkit) — judge-vs-human calibration (Cohen's κ / Kendall-τ vs Landis–Koch bands)
- [**agentic-eval-harness**](https://github.com/Mike-E-Log/agentic-eval-harness) — eval-gated Claude Code phase boundaries with cross-vendor scorecards
- [**ai-eval-atlas**](https://github.com/Mike-E-Log/ai-eval-atlas) — practitioner + technique map, source-linked
- **ai-engineer-best-practices** *(you are here)* — handbook + `score` MCP tool (3-vendor judge ensemble)
- [**learn-ai-eval**](https://github.com/Mike-E-Log/learn-ai-eval) — Claude-tutored learning engine for the eval canon
Profile: [github.com/Mike-E-Log](https://github.com/Mike-E-Log) · website: [mikeilog.com](https://mikeilog.com)
