Content
<p align="center">
<img src="assets/aae-banner.svg" alt="Awesome Agent Engineering" width="820" />
</p>
<h1 align="center">Awesome Agent Engineering</h1>
> A long-term, continuously maintained knowledge base for **Agent Engineering** — the design, evaluation, optimization, benchmarking, and deployment of AI agents.
>
> 🎯 **This list is organized around the Evaluation Layer.** Evaluation is the highest-leverage activity in agent engineering — it is the only layer that turns guesswork into measurable, iteratively-improving systems. It is deliberately placed first.
**English** · [中文](README.zh-CN.md)
Stats are fetched live via `gh api`; each category holds **≤ 10 repos** (tier-aware, by stars). Last updated 2026-07-16.
## Acknowledgments
Built with the help of two communities that keep agent engineering practical, not just theoretical:
- **[Telegram OpenClaw 中文社区](https://t.me/OpenClaw_Group)** — the OpenClaw Chinese-speaking community on Telegram.
- **[Linux.do](https://linux.do)** — a Chinese developer / tech community (Discourse, trust-level based).
## How to use this repo
1. **Start with the Evaluation Layer** (Section 1) and its flagship depth page [`docs/evals.md`](docs/evals.md).
2. Then read the agent **Frameworks** (§2) and **Harness** (§3) — the runtime substrate.
3. Then the surfaces: **MCP / Tools / Browser / Coding** (§9–12).
4. Then **Infra** (§15), **Observability** (§7), **Memory** (§8) — the supporting layers.
5. Deep dives live in [`docs/`](docs/) — one page per domain, each with a Mermaid diagram and a *when-to-use-which* guide.
**Who it's for:** backend/ML engineers building agents, platform & SRE teams operating them, app builders, and researchers benchmarking them.
## Table of Contents
### Categories
- **1. [Evaluation Layer ★](#1-evaluation-layer--focus)** — Prompt Evals · Agent Eval · EDD *(focus)*
- 2. [Agent Frameworks](#2-agent-frameworks)
- 3. [Agent Harness](#3-agent-harness)
- 4. [Prompt Engineering](#4-prompt-engineering)
- 5. [LLM Benchmark](#5-llm-benchmark)
- 6. [Agent Benchmark](#6-agent-benchmark)
- 7. [Observability](#7-observability)
- 8. [Memory Systems](#8-memory-systems)
- 9. [MCP Ecosystem](#9-mcp-ecosystem)
- 10. [Tool Calling](#10-tool-calling)
- 11. [Browser Agents](#11-browser-agents)
- 12. [Coding Agents](#12-coding-agents)
- 13. [RAG](#13-rag)
- 14. [Workflow Engines](#14-workflow-engines)
- 15. [AI Infrastructure](#15-ai-infrastructure)
- 16. [Agent Safety](#16-agent-safety)
- 17. [Agent Optimization](#17-agent-optimization)
- 18. [Research Collections](#18-research-collections)
- 19. [Awesome Lists](#19-awesome-lists)
### Overviews
- [Cross-cutting Comparison Tables](#cross-cutting-comparison-tables)
- [Agent Engineering Architecture Stack (Mermaid)](#agent-engineering-architecture-stack)
- [End-to-End Agent + Eval Flow (Mermaid)](#end-to-end-agent--eval-flow)
- [Recommended Learning Order](#recommended-learning-order)
- [Learning Roadmap by Role](#learning-roadmap-by-role)
- [Yearly Update — 2026](#yearly-update--2026)
- [Contributing](#contributing) · [License](#license)
### Deep-dive pages (`docs/`)
- [`docs/evals.md`](docs/evals.md) — Evaluation Layer ★ ★ flagship
- [`docs/benchmark.md`](docs/benchmark.md) — Benchmarks (LLM + Agent)
- [`docs/prompting.md`](docs/prompting.md) — Prompt Engineering
- [`docs/harness.md`](docs/harness.md) — Agent Harness & Workflow Runtimes
- [`docs/observability.md`](docs/observability.md) — Observability
- [`docs/memory.md`](docs/memory.md) — Memory Systems
- [`docs/browser-agents.md`](docs/browser-agents.md) — Browser Agents
- [`docs/coding-agents.md`](docs/coding-agents.md) — Coding Agents
- [`docs/mcp.md`](docs/mcp.md) — MCP Ecosystem
- [`docs/optimization.md`](docs/optimization.md) — Agent Optimization
## 1. Evaluation Layer ★ <a id="eval-layer"></a>
Evaluation is the leverage point of agent engineering: regression tests catch prompt regressions before users do, A/B compares variants, trajectory/multi-turn scoring measures real agent behavior, and Evaluation-Driven Development (EDD) closes the loop into CI/CD-for-AI. This section merges **Prompt Evals**, **Agent Eval**, and **EDD** because they are one discipline. Depth: [`docs/evals.md`](docs/evals.md).
### 1.1 Prompt Evaluation
Evaluating and comparing *prompts* — regression testing, A/B, version comparison, pass/fail.
| Repo | ⭐ | Forks | Last commit | Activity | Priority | Difficulty | Summary |
|------|----|------|-------------|----------|----------|------------|---------|
| [langfuse/langfuse](https://github.com/langfuse/langfuse) | 31197 | 3289 | 2026-07-15 | Active | 5 | Intermediate | Open-source AI engineering platform: evals, observability, prompt mgmt. |
| [promptfoo/promptfoo](https://github.com/promptfoo/promptfoo) | 23284 | 2088 | 2026-07-15 | Active | 5 | Beginner | Declarative prompt/agent/RAG testing and red-teaming CLI. |
| [openai/evals](https://github.com/openai/evals) | 18922 | 3025 | 2026-04-14 | Moderate | 5 | Intermediate | Framework and open registry for evaluating LLMs and LLM systems. |
| [mlflow/mlflow](https://github.com/mlflow/mlflow) | 27040 | 6001 | 2026-07-15 | Active | 4 | Intermediate | Open-source AI engineering platform for agents, LLMs, and ML models. |
| [comet-ml/opik](https://github.com/comet-ml/opik) | 20608 | 1607 | 2026-07-15 | Active | 4 | Intermediate | Debug, evaluate, and monitor LLM apps with tracing and dashboards. |
| [confident-ai/deepeval](https://github.com/confident-ai/deepeval) | 16869 | 1661 | 2026-07-14 | Active | 4 | Intermediate | The LLM Evaluation Framework. |
| [arize-ai/phoenix](https://github.com/arize-ai/phoenix) | 10571 | 986 | 2026-07-15 | Active | 4 | Advanced | AI observability and evaluation. |
| [agenta-ai/agenta](https://github.com/agenta-ai/agenta) | 4295 | 568 | 2026-07-15 | Active | 4 | Intermediate | Open-source LLMOps platform: prompt playground, management, evaluation. |
| [truera/trulens](https://github.com/truera/trulens) | 3438 | 310 | 2026-07-11 | Active | 3 | Intermediate | Evaluation and tracking for LLM experiments and AI agents. |
| [hegelai/prompttools](https://github.com/hegelai/prompttools) | 3041 | 256 | 2026-02-11 | Moderate | 3 | Beginner | Open-source tools for prompt testing and experimentation. |
### 1.2 Agent Evaluation
Evaluating full *agents* — tool-use, multi-turn, trajectories, planning.
| Repo | ⭐ | Forks | Last commit | Activity | Priority | Difficulty | Summary |
|------|----|------|-------------|----------|----------|------------|---------|
| [promptfoo/promptfoo](https://github.com/promptfoo/promptfoo) | 23285 | 2088 | 2026-07-15 | Active | 5 | Intermediate | Open-source LLM prompt, agent, and red-teaming evaluation platform. |
| [comet-ml/opik](https://github.com/comet-ml/opik) | 20608 | 1607 | 2026-07-15 | Active | 5 | Intermediate | Debug, evaluate, and monitor LLM applications, RAG systems, and agentic workflows. |
| [openai/evals](https://github.com/openai/evals) | 18922 | 3025 | 2026-04-14 | Moderate | 5 | Intermediate | OpenAI's framework and registry for evaluating LLMs and LLM systems. |
| [confident-ai/deepeval](https://github.com/confident-ai/deepeval) | 16869 | 1661 | 2026-07-14 | Active | 5 | Intermediate | Open-source LLM evaluation framework with built-in metrics and test suites. |
| [explodinggradients/ragas](https://github.com/explodinggradients/ragas) | 14850 | 1564 | 2026-02-24 | Moderate | 4 | Intermediate | Framework for evaluating RAG and LLM applications with reference-free metrics. |
| [EleutherAI/lm-evaluation-harness](https://github.com/EleutherAI/lm-evaluation-harness) | 13289 | 3409 | 2026-07-13 | Active | 4 | Advanced | Framework for few-shot evaluation of language models on hundreds of benchmarks. |
| [arize-ai/phoenix](https://github.com/arize-ai/phoenix) | 10571 | 986 | 2026-07-15 | Active | 4 | Intermediate | AI observability and evaluation platform for LLM, agent, and computer-vision applications. |
| [SWE-bench/SWE-bench](https://github.com/SWE-bench/SWE-bench) | 5427 | 922 | 2026-04-01 | Moderate | 4 | Advanced | Benchmark for evaluating LLMs on real-world GitHub issue resolution. |
| [UKGovernmentBEIS/inspect_ai](https://github.com/UKGovernmentBEIS/inspect_ai) | 2352 | 603 | 2026-07-15 | Active | 4 | Advanced | Framework for large language model evaluations from the UK AI Safety Institute. |
| [langchain-ai/langsmith-sdk](https://github.com/langchain-ai/langsmith-sdk) | 968 | 260 | 2026-07-15 | Active | 4 | Intermediate | LangSmith client SDK for tracing, evaluating, and monitoring LangChain applications. |
### 1.3 Evaluation-Driven Development (EDD)
Iterative improvement through automated evaluation loops — regression, experiment tracking, CI/CD-for-AI.
| Repo | ⭐ | Forks | Last commit | Activity | Priority | Difficulty | Summary |
|------|----|------|-------------|----------|----------|------------|---------|
| [langfuse/langfuse](https://github.com/langfuse/langfuse) | 31200 | 3289 | 2026-07-15 | Active | 5 | Intermediate | Open-source AI engineering platform for LLM evals, observability, metrics, prompt management, and datasets. |
| [promptfoo/promptfoo](https://github.com/promptfoo/promptfoo) | 23285 | 2088 | 2026-07-15 | Active | 5 | Intermediate | Test prompts, agents, and RAGs with red teaming, pentesting, and CI/CD integration. |
| [confident-ai/deepeval](https://github.com/confident-ai/deepeval) | 16868 | 1661 | 2026-07-14 | Active | 5 | Intermediate | The LLM Evaluation Framework with built-in metrics and CI/CD integration. |
| [EleutherAI/lm-evaluation-harness](https://github.com/EleutherAI/lm-evaluation-harness) | 13289 | 3409 | 2026-07-13 | Active | 5 | Advanced | A framework for few-shot evaluation of language models. |
| [mlflow/mlflow](https://github.com/mlflow/mlflow) | 27040 | 6001 | 2026-07-15 | Active | 4 | Intermediate | Open-source AI engineering platform for debugging, evaluating, monitoring, and optimizing AI applications. |
| [comet-ml/opik](https://github.com/comet-ml/opik) | 20608 | 1607 | 2026-07-15 | Active | 4 | Intermediate | Debug, evaluate, and monitor LLM applications, RAG systems, and agentic workflows. |
| [iterative/dvc](https://github.com/iterative/dvc) | 15750 | 1313 | 2026-07-14 | Active | 4 | Intermediate | Data versioning and ML experiments for reproducible AI development. |
| [arize-ai/phoenix](https://github.com/arize-ai/phoenix) | 10571 | 986 | 2026-07-15 | Active | 4 | Intermediate | AI observability and evaluation platform for LLM and agent applications. |
| [traceloop/openllmetry](https://github.com/traceloop/openllmetry) | 7302 | 1018 | 2026-07-13 | Active | 4 | Intermediate | Open-source observability for GenAI and LLM applications based on OpenTelemetry. |
| [helicone/helicone](https://github.com/helicone/helicone) | 5948 | 627 | 2026-07-05 | Active | 4 | Beginner | Open-source LLM observability platform with one-line integration for monitoring and evaluation. |
#### Why the Evaluation Layer comes first
Agents fail silently: a prompt tweak can drop tool-call accuracy 20% with no visible error. The Evaluation Layer is the only subsystem that makes agent behavior *measurable* and *iteratively improvable* rather than anecdotal. Every other layer (framework, harness, memory, infra) is only as good as the feedback loop that grades it — so evals come first, both in this list and in any mature agent stack.
```mermaid
flowchart LR
DS[(Datasets /<br/>Eval Cases)] --> RUN[Prompt /<br/>Agent Run]
RUN --> TR[Trace Capture]
TR --> AS[Assertions +<br/>LLM Judge]
AS --> SC[Aggregated Score]
SC --> RG{Regression<br/>Gate}
SC --> AB{A/B Compare}
SC --> CI{CI/CD<br/>for AI}
RG -. ship .-> SHP[✅ Ship]
RG -. regress .-> IT[🔁 Iterate]
AB --> IT
CI --> IT
IT --> RUN
```
## 2. Agent Frameworks
Top pick: [langflow-ai/langflow](https://github.com/langflow-ai/langflow) — Visual framework for building and deploying AI-powered agents and workflows.
| Repo | ⭐ | Forks | Last commit | Activity | Priority | Difficulty | Summary |
|------|----|------|-------------|----------|----------|------------|---------|
| [langflow-ai/langflow](https://github.com/langflow-ai/langflow) | 151902 | 9675 | 2026-07-15 | Active | 5 | Beginner | Visual framework for building and deploying AI-powered agents and workflows. |
| [langchain-ai/langchain](https://github.com/langchain-ai/langchain) | 141831 | 23563 | 2026-07-15 | Active | 5 | Intermediate | The agent engineering platform — tools, chains, and agents for LLM applications. |
| [modelcontextprotocol/servers](https://github.com/modelcontextprotocol/servers) | 88505 | 11224 | 2026-07-10 | Active | 5 | Intermediate | Reference servers and protocol examples for MCP. |
| [All-Hands-AI/OpenHands](https://github.com/All-Hands-AI/OpenHands) | 80858 | 10324 | 2026-07-15 | Active | 5 | Advanced | AI-driven software development agent platform. |
| [microsoft/autogen](https://github.com/microsoft/autogen) | 59748 | 8996 | 2026-04-15 | Moderate | 5 | Advanced | A programming framework for agentic AI. |
| [crewAIInc/crewAI](https://github.com/crewAIInc/crewAI) | 55563 | 7837 | 2026-07-15 | Active | 5 | Intermediate | Framework for orchestrating role-playing, autonomous AI agents. |
| [langchain-ai/langgraph](https://github.com/langchain-ai/langgraph) | 37349 | 6257 | 2026-07-15 | Active | 5 | Intermediate | Build resilient agents with graph-based state machines. |
| [openai/openai-agents-python](https://github.com/openai/openai-agents-python) | 27926 | 4329 | 2026-07-15 | Active | 5 | Intermediate | Lightweight, powerful framework for multi-agent workflows. |
| [modelcontextprotocol/python-sdk](https://github.com/modelcontextprotocol/python-sdk) | 23617 | 3645 | 2026-07-15 | Active | 5 | Intermediate | Official Python SDK for Model Context Protocol. |
| [run-llama/llama_index](https://github.com/run-llama/llama_index) | 50865 | 7754 | 2026-07-13 | Active | 4 | Intermediate | Leading document agent and OCR platform. |
## 3. Agent Harness
Top pick: [langchain-ai/langgraph](https://github.com/langchain-ai/langgraph) — Graph-based execution harness for building stateful, multi-step agent workflows with retries, branching, and persistence.
| Repo | ⭐ | Forks | Last commit | Activity | Priority | Difficulty | Summary |
|------|----|------|-------------|----------|----------|------------|---------|
| [langchain-ai/langgraph](https://github.com/langchain-ai/langgraph) | 37349 | 6257 | 2026-07-15 | Active | 5 | Intermediate | Graph-based execution harness for building stateful, multi-step agent workflows with retries, branching, and persistence. |
| [openai/openai-agents-python](https://github.com/openai/openai-agents-python) | 27926 | 4329 | 2026-07-15 | Active | 5 | Beginner | Lightweight framework for multi-agent workflows with tracing, handoffs, and guardrails. |
| [temporalio/temporal](https://github.com/temporalio/temporal) | 21655 | 1735 | 2026-07-15 | Active | 5 | Advanced | Durable execution platform for long-running workflows with automatic retries, state persistence, and failure recovery. |
| [mem0ai/mem0](https://github.com/mem0ai/mem0) | 60891 | 7093 | 2026-07-15 | Active | 4 | Intermediate | Universal memory layer for AI agents, enabling long-term state and personalization across runs. |
| [microsoft/autogen](https://github.com/microsoft/autogen) | 59748 | 8996 | 2026-04-15 | Active | 4 | Intermediate | Programming framework for agentic AI with multi-agent conversation patterns and code execution. |
| [crewAIInc/crewAI](https://github.com/crewAIInc/crewAI) | 55563 | 7837 | 2026-07-15 | Active | 4 | Intermediate | Framework for orchestrating role-playing autonomous AI agents into collaborative crews. |
| [google/adk-python](https://github.com/google/adk-python) | 20620 | 3695 | 2026-07-15 | Active | 4 | Intermediate | Google's open-source Python toolkit for building, evaluating, and deploying AI agents. |
| [pydantic/pydantic-ai](https://github.com/pydantic/pydantic-ai) | 18544 | 2366 | 2026-07-15 | Active | 4 | Intermediate | Type-safe agent framework with structured outputs, dependency injection, and runtime evaluation. |
| [camel-ai/camel](https://github.com/camel-ai/camel) | 17396 | 2000 | 2026-07-14 | Active | 4 | Advanced | Multi-agent framework focused on communicative agents and scaling laws of agent societies. |
| [inngest/inngest](https://github.com/inngest/inngest) | 5604 | 326 | 2026-07-15 | Active | 4 | Intermediate | Workflow orchestration platform for stateful step functions and AI workflows on serverless, servers, or edge. |
## 4. Prompt Engineering
Top pick: [stanfordnlp/dsp](https://github.com/stanfordnlp/dsp) — DSPy: system for programming language models by writing declarative modules.
| Repo | ⭐ | Forks | Last commit | Activity | Priority | Difficulty | Summary |
|------|----|------|-------------|----------|----------|------------|---------|
| [stanfordnlp/dsp](https://github.com/stanfordnlp/dsp) | 36148 | 3102 | 2026-07-15 | Active | 5 | Advanced | DSPy: system for programming language models by writing declarative modules. |
| [langfuse/langfuse](https://github.com/langfuse/langfuse) | 31197 | 3289 | 2026-07-15 | Active | 5 | Intermediate | Open-source AI engineering platform with prompt management, evaluations, and observability. |
| [promptfoo/promptfoo](https://github.com/promptfoo/promptfoo) | 23284 | 2088 | 2026-07-15 | Active | 5 | Beginner | Declarative prompt/agent evaluation and red-teaming framework with CI/CD integration. |
| [microsoft/guidance](https://github.com/microsoft/guidance) | 21667 | 1193 | 2026-05-21 | Active | 5 | Advanced | Microsoft DSL for controlling LLM generation via a guidance language. |
| [outlines-dev/outlines](https://github.com/outlines-dev/outlines) | 14508 | 777 | 2026-07-13 | Active | 4 | Advanced | Fast structured generation and constrained decoding for LLMs. |
| [jxnl/instructor](https://github.com/jxnl/instructor) | 13535 | 1154 | 2026-07-13 | Active | 4 | Intermediate | Python library for structured outputs from LLMs using Pydantic. |
| [microsoft/promptflow](https://github.com/microsoft/promptflow) | 11183 | 1111 | 2026-07-09 | Active | 4 | Intermediate | End-to-end LLM application development and lifecycle management tool. |
| [microsoft/TypeChat](https://github.com/microsoft/TypeChat) | 8673 | 415 | 2026-07-07 | Active | 4 | Intermediate | Type-driven natural language interfaces for typed LLM interactions. |
| [BoundaryML/baml](https://github.com/BoundaryML/baml) | 8544 | 448 | 2026-07-15 | Active | 4 | Advanced | Programming language and runtime for building reliable LLM-powered applications. |
| [pezzolabs/pezzo](https://github.com/pezzolabs/pezzo) | 3255 | 276 | 2026-03-31 | Moderate | 4 | Intermediate | Open-source LLMOps platform focused on prompt design, versioning, and collaboration. |
## 5. LLM Benchmark
Top pick: [EleutherAI/lm-evaluation-harness](https://github.com/EleutherAI/lm-evaluation-harness) — Widely-used few-shot evaluation harness for language models.
| Repo | ⭐ | Forks | Last commit | Activity | Priority | Difficulty | Summary |
|------|----|------|-------------|----------|----------|------------|---------|
| [EleutherAI/lm-evaluation-harness](https://github.com/EleutherAI/lm-evaluation-harness) | 13289 | 3409 | 2026-07-13 | Active | 5 | Intermediate | Widely-used few-shot evaluation harness for language models. |
| [princeton-nlp/SWE-bench](https://github.com/princeton-nlp/SWE-bench) | 5427 | 922 | 2026-04-01 | Moderate | 5 | Advanced | Benchmark for evaluating LLMs on resolving real-world GitHub issues. |
| [openai/human-eval](https://github.com/openai/human-eval) | 3301 | 451 | 2025-01-17 | Stale | 5 | Beginner | Original benchmark and dataset for evaluating code generation from docstrings. |
| [stanford-crfm/helm](https://github.com/stanford-crfm/helm) | 2855 | 404 | 2026-07-01 | Active | 5 | Intermediate | HELM provides a multi-metric, scenario-based evaluation framework that covers accuracy, calibration, robustness, efficiency, bias, and toxicity. |
| [aider-ai/aider](https://github.com/aider-ai/aider) | 47399 | 4733 | 2026-05-22 | Active | 4 | Advanced | AI pair programming tool with extensive benchmark support for code editing. |
| [lm-sys/FastChat](https://github.com/lm-sys/FastChat) | 39490 | 4785 | 2026-05-01 | Active | 4 | Intermediate | Open platform for training, serving, and evaluating LLMs, including Chatbot Arena. |
| [stanfordnlp/dspy](https://github.com/stanfordnlp/dspy) | 36148 | 3102 | 2026-07-15 | Active | 4 | Intermediate | Framework for programming language models with optimizers and evaluation primitives. |
| [promptfoo/promptfoo](https://github.com/promptfoo/promptfoo) | 23284 | 2088 | 2026-07-15 | Active | 4 | Beginner | Open-source evaluation and red-teaming framework for LLM prompts, agents, and RAG. |
| [openai/evals](https://github.com/openai/evals) | 18922 | 3025 | 2026-04-14 | Moderate | 4 | Intermediate | Open-source registry of benchmarks and an evaluation framework for LLMs and LLM systems. |
| [OpenRLHF/OpenRLHF](https://github.com/OpenRLHF/OpenRLHF) | 9795 | 988 | 2026-07-14 | Active | 4 | Advanced | Scalable RLHF framework with evaluation utilities for aligned models. |
## 6. Agent Benchmark
Top pick: [princeton-nlp/SWE-agent](https://github.com/princeton-nlp/SWE-agent) — An autonomous agent that fixes GitHub issues end-to-end using any language model.
| Repo | ⭐ | Forks | Last commit | Activity | Priority | Difficulty | Summary |
|------|----|------|-------------|----------|----------|------------|---------|
| [princeton-nlp/SWE-agent](https://github.com/princeton-nlp/SWE-agent) | 19818 | 2167 | 2026-07-13 | Active | 5 | Advanced | An autonomous agent that fixes GitHub issues end-to-end using any language model. |
| [ShishirPatil/gorilla](https://github.com/ShishirPatil/gorilla) | 12948 | 1389 | 2026-04-13 | Moderate | 5 | Advanced | Training and evaluation framework for large language models on API and tool calling. |
| [OpenBMB/ToolBench](https://github.com/OpenBMB/ToolBench) | 5697 | 486 | 2025-05-21 | Stale | 5 | Advanced | Open platform for training, serving, and evaluating large language models for tool learning. |
| [princeton-nlp/SWE-bench](https://github.com/princeton-nlp/SWE-bench) | 5427 | 922 | 2026-04-01 | Moderate | 5 | Advanced | Benchmark that evaluates whether language models can resolve real-world GitHub issues. |
| [xlang-ai/OSWorld](https://github.com/xlang-ai/OSWorld) | 3013 | 494 | 2026-07-15 | Active | 5 | Advanced | Benchmark for multimodal agents solving open-ended tasks in real computer environments. |
| [harbor-framework/terminal-bench](https://github.com/harbor-framework/terminal-bench) | 2454 | 557 | 2026-07-11 | Active | 5 | Advanced | A benchmark for LLMs on complex tasks in the terminal. |
| [openai/simple-evals](https://github.com/openai/simple-evals) | 4570 | 495 | 2026-04-22 | Moderate | 3 | Beginner | Lightweight evaluation framework and registry for LLMs and LLM systems. |
| [THUDM/AgentBench](https://github.com/THUDM/AgentBench) | 3575 | 270 | 2026-02-08 | Moderate | 4 | Advanced | Comprehensive benchmark to evaluate LLMs as agents across multiple environments. |
| [sierra-research/tau-bench](https://github.com/sierra-research/tau-bench) | 1328 | 209 | 2026-03-18 | Moderate | 4 | Intermediate | Benchmark and code for tau-bench, focusing on tool-augmented task agents. |
| [OSU-NLP-Group/Mind2Web](https://github.com/OSU-NLP-Group/Mind2Web) | 1015 | 123 | 2025-11-05 | Moderate | 4 | Intermediate | Dataset and benchmark for generalist web agents based on large language models. |
## 7. Observability
Top pick: [grafana/grafana](https://github.com/grafana/grafana) — Open observability and data visualization platform.
| Repo | ⭐ | Forks | Last commit | Activity | Priority | Difficulty | Summary |
|------|----|------|-------------|----------|----------|------------|---------|
| [grafana/grafana](https://github.com/grafana/grafana) | 75647 | 14361 | 2026-07-15 | Active | 5 | Intermediate | Open observability and data visualization platform. |
| [prometheus/prometheus](https://github.com/prometheus/prometheus) | 65231 | 10753 | 2026-07-15 | Active | 5 | Advanced | Open-source monitoring system and time series database. |
| [langfuse/langfuse](https://github.com/langfuse/langfuse) | 31197 | 3289 | 2026-07-15 | Active | 5 | Intermediate | Open-source AI engineering platform for LLM observability, evals and prompt management. |
| [mlflow/mlflow](https://github.com/mlflow/mlflow) | 27040 | 6001 | 2026-07-15 | Active | 5 | Intermediate | Open-source AI/ML lifecycle and observability platform. |
| [comet-ml/opik](https://github.com/comet-ml/opik) | 20608 | 1607 | 2026-07-15 | Active | 5 | Intermediate | Open-source observability, evaluation and monitoring for LLM, RAG and agent workflows. |
| [confident-ai/deepeval](https://github.com/confident-ai/deepeval) | 16869 | 1661 | 2026-07-14 | Active | 4 | Intermediate | LLM evaluation framework. |
| [explodinggradients/ragas](https://github.com/explodinggradients/ragas) | 14850 | 1564 | 2026-02-24 | Moderate | 4 | Intermediate | Evaluation framework for RAG and LLM applications. |
| [arize-ai/phoenix](https://github.com/arize-ai/phoenix) | 10571 | 986 | 2026-07-15 | Active | 4 | Intermediate | Open-source AI observability and evaluation platform. |
| [traceloop/openllmetry](https://github.com/traceloop/openllmetry) | 7302 | 1018 | 2026-07-13 | Active | 4 | Intermediate | OpenTelemetry-based observability instrumentation for GenAI/LLM apps. |
| [open-telemetry/opentelemetry-go](https://github.com/open-telemetry/opentelemetry-go) | 6465 | 1417 | 2026-07-15 | Active | 4 | Intermediate | Official OpenTelemetry Go API and SDK. |
## 8. Memory Systems
Top pick: [mem0ai/mem0](https://github.com/mem0ai/mem0) — Universal memory layer for AI agents with user/professional identity, adaptive personalization, and broad framework integrations.
| Repo | ⭐ | Forks | Last commit | Activity | Priority | Difficulty | Summary |
|------|----|------|-------------|----------|----------|------------|---------|
| [mem0ai/mem0](https://github.com/mem0ai/mem0) | 60891 | 7093 | 2026-07-15 | Active | 5 | Intermediate | Universal memory layer for AI agents with user/professional identity, adaptive personalization, and broad framework integrations. |
| [topoteretes/cognee](https://github.com/topoteretes/cognee) | 27933 | 2760 | 2026-07-15 | Active | 5 | Intermediate | Open-source self-hosted AI memory platform that builds a knowledge graph from conversations and documents for persistent agent memory. |
| [letta-ai/letta](https://github.com/letta-ai/letta) | 23804 | 2521 | 2026-07-03 | Active | 5 | Advanced | Stateful agent platform (formerly MemGPT) with advanced memory management, self-editing memory, and long-context reasoning. |
| [getzep/graphiti](https://github.com/getzep/graphiti) | 28750 | 2903 | 2026-07-15 | Active | 4 | Intermediate | Real-time knowledge graph building library designed for agent memory and RAG. |
| [supermemoryai/supermemory](https://github.com/supermemoryai/supermemory) | 28387 | 2472 | 2026-07-14 | Active | 4 | Intermediate | Fast, scalable, locally-runnable memory and context engine with a memory API and web app. |
| [rohitg00/agentmemory](https://github.com/rohitg00/agentmemory) | 25166 | 2079 | 2026-07-13 | Active | 4 | Beginner | Persistent memory for AI coding agents, ranked #1 on real-world benchmarks. |
| [TencentCloud/TencentDB-Agent-Memory](https://github.com/TencentCloud/TencentDB-Agent-Memory) | 8933 | 820 | 2026-07-14 | Active | 3 | Intermediate | Fully local long-term memory for AI agents using a 4-tier progressive pipeline with no external API dependencies. |
| [getzep/zep](https://github.com/getzep/zep) | 4754 | 640 | 2026-07-10 | Active | 4 | Intermediate | Long-term memory store and platform for LLM applications with entity extraction, summarization, and classification. |
| [langchain-ai/langmem](https://github.com/langchain-ai/langmem) | 1562 | 176 | 2026-07-15 | Active | 4 | Intermediate | LangChain library for adding long-term memory to agents with built-in memory management tools. |
| [NirDiamant/Agent_Memory_Techniques](https://github.com/NirDiamant/Agent_Memory_Techniques) | 788 | 104 | 2026-07-14 | Active | 4 | Beginner | Educational collection of 30 runnable notebooks covering buffers, vector stores, knowledge graphs, episodic/semantic memory, and production patterns. |
## 9. MCP Ecosystem
Top pick: [modelcontextprotocol/servers](https://github.com/modelcontextprotocol/servers) — Official reference implementations of MCP servers maintained by Anthropic.
| Repo | ⭐ | Forks | Last commit | Activity | Priority | Difficulty | Summary |
|------|----|------|-------------|----------|----------|------------|---------|
| [modelcontextprotocol/servers](https://github.com/modelcontextprotocol/servers) | 88505 | 11224 | 2026-07-10 | Active | 5 | Intermediate | Official reference implementations of MCP servers maintained by Anthropic. |
| [jlowin/fastmcp](https://github.com/jlowin/fastmcp) | 26217 | 2133 | 2026-07-14 | Active | 5 | Beginner | High-level, ergonomic Python framework for building MCP servers quickly. |
| [modelcontextprotocol/python-sdk](https://github.com/modelcontextprotocol/python-sdk) | 23617 | 3645 | 2026-07-15 | Active | 5 | Beginner | Official Python SDK for building MCP servers and clients. |
| [modelcontextprotocol/typescript-sdk](https://github.com/modelcontextprotocol/typescript-sdk) | 12860 | 1982 | 2026-07-15 | Active | 5 | Beginner | Official TypeScript SDK for building MCP servers and clients. |
| [modelcontextprotocol/modelcontextprotocol](https://github.com/modelcontextprotocol/modelcontextprotocol) | 8599 | 1650 | 2026-07-15 | Active | 5 | Intermediate | Specification and documentation for the Model Context Protocol. |
| [punkpeye/awesome-mcp-servers](https://github.com/punkpeye/awesome-mcp-servers) | 90809 | 13019 | 2026-07-13 | Active | 4 | Beginner | Curated list of MCP servers and resources. |
| [cline/cline](https://github.com/cline/cline) | 64683 | 6917 | 2026-07-15 | Active | 4 | Intermediate | Autonomous coding agent for VS Code with MCP support. |
| [microsoft/playwright-mcp](https://github.com/microsoft/playwright-mcp) | 35112 | 2922 | 2026-07-15 | Active | 4 | Intermediate | Microsoft's official browser automation MCP server using Playwright. |
| [github/github-mcp-server](https://github.com/github/github-mcp-server) | 31463 | 4584 | 2026-07-15 | Active | 4 | Intermediate | GitHub's official MCP server for repository, issue, PR, and search operations. |
| [tadata-org/fastapi_mcp](https://github.com/tadata-org/fastapi_mcp) | 11944 | 952 | 2025-11-24 | Moderate | 4 | Beginner | Expose FastAPI endpoints as MCP tools. |
## 10. Tool Calling
Top pick: [denoland/deno](https://github.com/denoland/deno) — A modern runtime for JavaScript and TypeScript with built-in sandboxing, permissions, and secure code execution.
| Repo | ⭐ | Forks | Last commit | Activity | Priority | Difficulty | Summary |
|------|----|------|-------------|----------|----------|------------|---------|
| [denoland/deno](https://github.com/denoland/deno) | 107811 | 6283 | 2026-07-15 | Active | 5 | Intermediate | A modern runtime for JavaScript and TypeScript with built-in sandboxing, permissions, and secure code execution. |
| [composiodev/composio](https://github.com/composiodev/composio) | 29243 | 4661 | 2026-07-15 | Active | 5 | Intermediate | Composio powers 1000+ toolkits, tool search, context management, authentication, and a sandboxed workbench for AI agents. |
| [openai/openai-agents-python](https://github.com/openai/openai-agents-python) | 27926 | 4329 | 2026-07-15 | Active | 5 | Intermediate | A lightweight, powerful framework for multi-agent workflows. |
| [modelcontextprotocol/python-sdk](https://github.com/modelcontextprotocol/python-sdk) | 23617 | 3645 | 2026-07-15 | Active | 5 | Intermediate | The official Python SDK for Model Context Protocol servers and clients. |
| [daytonaio/daytona](https://github.com/daytonaio/daytona) | 72256 | 5661 | 2026-07-09 | Active | 4 | Intermediate | Secure and elastic infrastructure for running AI-generated code. |
| [openai/swarm](https://github.com/openai/swarm) | 21798 | 2324 | 2026-04-15 | Moderate | 4 | Beginner | Educational framework exploring ergonomic, lightweight multi-agent orchestration. |
| [pyodide/pyodide](https://github.com/pyodide/pyodide) | 14736 | 1032 | 2026-07-15 | Active | 4 | Advanced | Python distribution for the browser and Node.js based on WebAssembly. |
| [e2b-dev/E2B](https://github.com/e2b-dev/E2B) | 12984 | 965 | 2026-07-15 | Active | 4 | Intermediate | Open-source, secure environment with real-world tools for enterprise-grade agents. |
| [modelcontextprotocol/specification](https://github.com/modelcontextprotocol/specification) | 8599 | 1650 | 2026-07-15 | Active | 4 | Intermediate | Specification and documentation for the Model Context Protocol. |
| [browserbase/mcp-server-browserbase](https://github.com/browserbase/mcp-server-browserbase) | 3406 | 363 | 2026-07-07 | Active | 4 | Intermediate | Allow LLMs to control a browser with Browserbase and Stagehand. |
## 11. Browser Agents
Top pick: [browser-use/browser-use](https://github.com/browser-use/browser-use) — 🌐 Make websites accessible for AI agents. Automate tasks online with ease.
| Repo | ⭐ | Forks | Last commit | Activity | Priority | Difficulty | Summary |
|------|----|------|-------------|----------|----------|------------|---------|
| [browser-use/browser-use](https://github.com/browser-use/browser-use) | 104843 | 11550 | 2026-07-15 | Active | 5 | Intermediate | 🌐 Make websites accessible for AI agents. Automate tasks online with ease. |
| [microsoft/playwright-mcp](https://github.com/microsoft/playwright-mcp) | 35112 | 2922 | 2026-07-15 | Active | 5 | Intermediate | Playwright MCP server |
| [browserbase/stagehand](https://github.com/browserbase/stagehand) | 23516 | 1616 | 2026-07-14 | Active | 5 | Intermediate | The SDK For Browser Agents |
| [Skyvern-AI/skyvern](https://github.com/Skyvern-AI/skyvern) | 22243 | 2084 | 2026-07-15 | Active | 4 | Intermediate | Automate browser based workflows with AI |
| [trycua/cua](https://github.com/trycua/cua) | 19706 | 1296 | 2026-07-15 | Active | 4 | Advanced | Open-source infrastructure for Computer-Use Agents. Sandboxes, SDKs, and benchmarks to train and evaluate AI agents that can control full desktops (macOS, Linux, Windows). |
| [browser-use/web-ui](https://github.com/browser-use/web-ui) | 16197 | 2715 | 2026-05-15 | Active | 4 | Beginner | 🖥️ Run AI Agent in your browser. |
| [web-infra-dev/midscene](https://github.com/web-infra-dev/midscene) | 14078 | 1072 | 2026-07-15 | Active | 4 | Intermediate | AI-powered, vision-driven UI automation for every platform. |
| [nanobrowser/nanobrowser](https://github.com/nanobrowser/nanobrowser) | 13467 | 1413 | 2025-11-24 | Moderate | 4 | Beginner | Open-Source Chrome extension for AI-powered web automation. Run multi-agent workflows using your own LLM API key. Alternative to OpenAI Operator. |
| [steel-dev/steel-browser](https://github.com/steel-dev/steel-browser) | 7339 | 948 | 2026-07-12 | Active | 4 | Intermediate | 🔥 Open Source Browser API for AI Agents & Apps. Steel Browser is a batteries-included browser sandbox that lets you automate the web without worrying about infrastructure. |
| [microsoft/Webwright](https://github.com/microsoft/Webwright) | 5808 | 367 | 2026-06-03 | Active | 4 | Advanced | A simple SWE style browser agent framework that achieves SOTA results on long horizon web tasks. |
## 12. Coding Agents
Top pick: [langchain-ai/langchain](https://github.com/langchain-ai/langchain) — The dominant agent engineering platform and framework.
| Repo | ⭐ | Forks | Last commit | Activity | Priority | Difficulty | Summary |
|------|----|------|-------------|----------|----------|------------|---------|
| [langchain-ai/langchain](https://github.com/langchain-ai/langchain) | 141832 | 23563 | 2026-07-15 | Active | 5 | Advanced | The dominant agent engineering platform and framework. |
| [anthropics/claude-code](https://github.com/anthropics/claude-code) | 137941 | 22246 | 2026-07-14 | Active | 5 | Intermediate | Anthropic's official agentic coding CLI. |
| [openai/codex](https://github.com/openai/codex) | 98364 | 14663 | 2026-07-15 | Active | 5 | Intermediate | OpenAI's official lightweight terminal coding agent. |
| [All-Hands-AI/OpenHands](https://github.com/All-Hands-AI/OpenHands) | 80859 | 10325 | 2026-07-15 | Active | 5 | Advanced | Open-source platform for AI software engineering agents. |
| [google-gemini/gemini-cli](https://github.com/google-gemini/gemini-cli) | 106001 | 14263 | 2026-07-15 | Active | 4 | Intermediate | Google's open-source terminal agent for the Gemini ecosystem. |
| [cline/cline](https://github.com/cline/cline) | 64683 | 6917 | 2026-07-15 | Active | 4 | Intermediate | Autonomous coding agent as VS Code extension / SDK / CLI. |
| [block/goose](https://github.com/block/goose) | 51235 | 5694 | 2026-07-15 | Active | 4 | Intermediate | Block's extensible open-source coding agent. |
| [Aider-AI/aider](https://github.com/Aider-AI/aider) | 47400 | 4733 | 2026-05-22 | Active | 4 | Intermediate | Terminal pair-programmer that edits files with your existing code. |
| [continuedev/continue](https://github.com/continuedev/continue) | 34889 | 5050 | 2026-07-15 | Active | 4 | Intermediate | Open-source autopilot for VS Code. |
| [TabbyML/tabby](https://github.com/TabbyML/tabby) | 33701 | 1768 | 2026-06-30 | Active | 3 | Intermediate | Self-hosted AI coding assistant. |
## 13. RAG
Top pick: [langchain-ai/langchain](https://github.com/langchain-ai/langchain) — The leading Python framework for building LLM applications with robust RAG tooling, vector stores, and agentic chains.
| Repo | ⭐ | Forks | Last commit | Activity | Priority | Difficulty | Summary |
|------|----|------|-------------|----------|----------|------------|---------|
| [langchain-ai/langchain](https://github.com/langchain-ai/langchain) | 141832 | 23563 | 2026-07-15 | Active | 5 | Intermediate | The leading Python framework for building LLM applications with robust RAG tooling, vector stores, and agentic chains. |
| [infiniflow/ragflow](https://github.com/infiniflow/ragflow) | 85105 | 9935 | 2026-07-15 | Active | 5 | Intermediate | Open-source RAG engine that fuses deep-document understanding with agentic retrieval and a visual workflow builder. |
| [run-llama/llama_index](https://github.com/run-llama/llama_index) | 50865 | 7754 | 2026-07-13 | Active | 5 | Intermediate | Document-agent and data-framework for RAG with rich connectors, indices, and agentic retrieval primitives. |
| [HKUDS/LightRAG](https://github.com/HKUDS/LightRAG) | 37699 | 5304 | 2026-07-15 | Active | 5 | Intermediate | Lightweight, fast GraphRAG implementation that combines graph and vector retrieval for cost-efficient knowledge-base QA. |
| [microsoft/graphrag](https://github.com/microsoft/graphrag) | 34443 | 3641 | 2026-07-15 | Active | 5 | Advanced | Microsoft's modular graph-based RAG system using community detection and global/local query modes. |
| [milvus-io/milvus](https://github.com/milvus-io/milvus) | 45234 | 4121 | 2026-07-15 | Active | 4 | Advanced | Cloud-native, distributed vector database for massive-scale ANN and RAG deployments. |
| [langchain-ai/langgraph](https://github.com/langchain-ai/langgraph) | 37349 | 6257 | 2026-07-15 | Active | 4 | Intermediate | Agent orchestration framework from LangChain for building resilient, stateful multi-agent and RAG systems. |
| [qdrant/qdrant](https://github.com/qdrant/qdrant) | 33296 | 2490 | 2026-07-15 | Active | 4 | Intermediate | High-performance Rust vector database with hybrid search, filtering, and cloud-native scaling. |
| [chroma-core/chroma](https://github.com/chroma-core/chroma) | 28795 | 2385 | 2026-07-14 | Active | 4 | Intermediate | Developer-friendly vector database focused on simplicity, embeddings, and local-first RAG workflows. |
| [deepset-ai/haystack](https://github.com/deepset-ai/haystack) | 25902 | 2923 | 2026-07-15 | Active | 4 | Intermediate | Production-ready AI orchestration framework for modular RAG, semantic search, and agent pipelines. |
## 14. Workflow Engines
Top pick: [n8n-io/n8n](https://github.com/n8n-io/n8n) — Fair-code workflow automation platform with native AI capabilities and 400+ integrations.
| Repo | ⭐ | Forks | Last commit | Activity | Priority | Difficulty | Summary |
|------|----|------|-------------|----------|----------|------------|---------|
| [n8n-io/n8n](https://github.com/n8n-io/n8n) | 196534 | 59331 | 2026-07-15 | Active | 5 | Beginner | Fair-code workflow automation platform with native AI capabilities and 400+ integrations. |
| [langflow-ai/langflow](https://github.com/langflow-ai/langflow) | 151902 | 9675 | 2026-07-15 | Active | 5 | Beginner | Visual tool for building and deploying AI-powered agents and workflows. |
| [langgenius/dify](https://github.com/langgenius/dify) | 148916 | 23443 | 2026-07-15 | Active | 5 | Intermediate | Production-ready open-source platform for building agentic AI workflows. |
| [apache/airflow](https://github.com/apache/airflow) | 46125 | 17390 | 2026-07-15 | Active | 5 | Intermediate | Programmatic authoring, scheduling, and monitoring of workflows. |
| [temporalio/temporal](https://github.com/temporalio/temporal) | 21655 | 1735 | 2026-07-15 | Active | 5 | Advanced | Durable execution engine for long-running workflows in Go and other languages. |
| [FlowiseAI/Flowise](https://github.com/FlowiseAI/Flowise) | 54648 | 24718 | 2026-07-15 | Active | 4 | Beginner | Open-source visual builder for AI agents and LLM workflows. |
| [kestra-io/kestra](https://github.com/kestra-io/kestra) | 27435 | 2820 | 2026-07-15 | Active | 4 | Intermediate | Event-driven orchestration and scheduling platform using YAML/blueprint definitions. |
| [PrefectHQ/prefect](https://github.com/PrefectHQ/prefect) | 23390 | 2410 | 2026-07-15 | Active | 4 | Intermediate | Modern Python workflow orchestration framework for resilient data pipelines. |
| [windmill-labs/windmill](https://github.com/windmill-labs/windmill) | 17163 | 1028 | 2026-07-15 | Active | 4 | Intermediate | Open-source developer platform and fast workflow engine with built-in webhooks and UIs. |
| [argoproj/argo-workflows](https://github.com/argoproj/argo-workflows) | 16826 | 3571 | 2026-07-15 | Active | 4 | Advanced | Cloud-native workflow engine for Kubernetes. |
## 15. AI Infrastructure
Top pick: [ollama/ollama](https://github.com/ollama/ollama) — Local LLM runtime and model management CLI/API.
| Repo | ⭐ | Forks | Last commit | Activity | Priority | Difficulty | Summary |
|------|----|------|-------------|----------|----------|------------|---------|
| [ollama/ollama](https://github.com/ollama/ollama) | 176156 | 16968 | 2026-07-15 | Active | 5 | Beginner | Local LLM runtime and model management CLI/API. |
| [ggerganov/llama.cpp](https://github.com/ggerganov/llama.cpp) | 120461 | 20600 | 2026-07-15 | Active | 5 | Advanced | High-performance LLM inference in C/C++ with broad quantization support. |
| [vllm-project/vllm](https://github.com/vllm-project/vllm) | 86324 | 19442 | 2026-07-15 | Active | 5 | Advanced | High-throughput, memory-efficient LLM serving engine. |
| [BerriAI/litellm](https://github.com/BerriAI/litellm) | 53671 | 9786 | 2026-07-15 | Active | 5 | Intermediate | OpenAI-compatible AI gateway and SDK for 100+ LLM APIs. |
| [sgl-project/sglang](https://github.com/sgl-project/sglang) | 30335 | 7171 | 2026-07-15 | Active | 5 | Advanced | High-performance serving framework for LLMs and multimodal models. |
| [mudler/LocalAI](https://github.com/mudler/LocalAI) | 47542 | 4237 | 2026-07-15 | Active | 4 | Intermediate | Open-source AI engine for running LLMs, vision, voice, and image models locally. |
| [ray-project/ray](https://github.com/ray-project/ray) | 43250 | 7794 | 2026-07-15 | Active | 4 | Advanced | Distributed AI compute engine and runtime for ML workloads. |
| [mlc-ai/mlc-llm](https://github.com/mlc-ai/mlc-llm) | 22949 | 2088 | 2026-07-13 | Active | 4 | Advanced | Universal LLM deployment engine with ML compilation. |
| [kvcache-ai/ktransformers](https://github.com/kvcache-ai/ktransformers) | 17437 | 1352 | 2026-07-15 | Active | 4 | Advanced | Heterogeneous LLM inference and fine-tune optimization framework. |
| [Portkey-AI/gateway](https://github.com/Portkey-AI/gateway) | 12429 | 1206 | 2026-05-25 | Active | 4 | Intermediate | Fast AI gateway with guardrails, routing to 1600+ models. |
## 16. Agent Safety
Top pick: [promptfoo/promptfoo](https://github.com/promptfoo/promptfoo) — Red teaming, pentesting, and vulnerability scanning for LLMs, agents, and RAGs.
| Repo | ⭐ | Forks | Last commit | Activity | Priority | Difficulty | Summary |
|------|----|------|-------------|----------|----------|------------|---------|
| [promptfoo/promptfoo](https://github.com/promptfoo/promptfoo) | 23285 | 2088 | 2026-07-15 | Active | 5 | Intermediate | Red teaming, pentesting, and vulnerability scanning for LLMs, agents, and RAGs. |
| [NVIDIA/garak](https://github.com/NVIDIA/garak) | 8446 | 1092 | 2026-07-14 | Active | 5 | Advanced | The LLM vulnerability scanner. |
| [guardrails-ai/guardrails](https://github.com/guardrails-ai/guardrails) | 7149 | 650 | 2026-07-14 | Active | 5 | Intermediate | Adding guardrails to large language models. |
| [NVIDIA/NeMo-Guardrails](https://github.com/NVIDIA/NeMo-Guardrails) | 6708 | 768 | 2026-07-15 | Active | 5 | Intermediate | Programmable guardrails for LLM-based conversational systems. |
| [microsoft/PyRIT](https://github.com/microsoft/PyRIT) | 4113 | 808 | 2026-07-15 | Active | 5 | Advanced | Python Risk Identification Tool for generative AI. |
| [microsoft/presidio](https://github.com/microsoft/presidio) | 10007 | 1202 | 2026-07-15 | Active | 4 | Intermediate | Open-source PII detection, redaction, masking, and anonymization framework. |
| [ibm/adversarial-robustness-toolbox](https://github.com/ibm/adversarial-robustness-toolbox) | 6108 | 1325 | 2025-12-12 | Moderate | 4 | Advanced | Python library for machine learning security — evasion, poisoning, extraction, inference. |
| [llm-attacks/llm-attacks](https://github.com/llm-attacks/llm-attacks) | 4739 | 630 | 2024-08-02 | Stale | 4 | Advanced | Universal and Transferable Attacks on Aligned Language Models. |
| [protectai/llm-guard](https://github.com/protectai/llm-guard) | 3184 | 423 | 2026-07-08 | Archived | 4 | Intermediate | The Security Toolkit for LLM Interactions. |
| [0din-ai/ai-scanner](https://github.com/0din-ai/ai-scanner) | 572 | 91 | 2026-07-14 | Active | 3 | Intermediate | AI model safety scanner built on NVIDIA garak. |
## 17. Agent Optimization
Top pick: [langchain-ai/langchain](https://github.com/langchain-ai/langchain) — The agent engineering platform for building applications with LLMs.
| Repo | ⭐ | Forks | Last commit | Activity | Priority | Difficulty | Summary |
|------|----|------|-------------|----------|----------|------------|---------|
| [langchain-ai/langchain](https://github.com/langchain-ai/langchain) | 141832 | 23563 | 2026-07-15 | Active | 5 | Intermediate | The agent engineering platform for building applications with LLMs. |
| [microsoft/autogen](https://github.com/microsoft/autogen) | 59748 | 8996 | 2026-04-15 | Moderate | 5 | Intermediate | Programming framework for agentic AI with conversational agents and optimization primitives. |
| [stanfordnlp/dspy](https://github.com/stanfordnlp/dspy) | 36148 | 3102 | 2026-07-15 | Active | 5 | Intermediate | Framework for programming—not prompting—language models via composable modules and optimizers. |
| [crewAIInc/crewAI](https://github.com/crewAIInc/crewAI) | 55565 | 7837 | 2026-07-15 | Active | 4 | Intermediate | Framework for orchestrating role-playing, autonomous AI agents. |
| [langfuse/langfuse](https://github.com/langfuse/langfuse) | 31200 | 3289 | 2026-07-15 | Active | 4 | Intermediate | Open-source AI engineering platform: LLM evals, observability, metrics, prompt management, playground, datasets. |
| [huggingface/smolagents](https://github.com/huggingface/smolagents) | 28366 | 2788 | 2026-07-14 | Active | 4 | Beginner | Barebones library for agents that think in code. |
| [openai/swarm](https://github.com/openai/swarm) | 21798 | 2324 | 2026-04-15 | Moderate | 3 | Beginner | Educational framework exploring lightweight multi-agent orchestration. |
| [openai/evals](https://github.com/openai/evals) | 18922 | 3025 | 2026-04-14 | Moderate | 4 | Intermediate | Framework for evaluating LLMs and LLM systems, plus an open-source benchmark registry. |
| [camel-ai/camel](https://github.com/camel-ai/camel) | 17397 | 2000 | 2026-07-14 | Active | 3 | Advanced | The first and the best multi-agent framework, focused on communicative agents and agent society scaling. |
| [confident-ai/deepeval](https://github.com/confident-ai/deepeval) | 16868 | 1661 | 2026-07-14 | Active | 4 | Intermediate | The LLM evaluation framework with built-in metrics for correctness, faithfulness, and more. |
## 18. Research Collections
Top pick: [dair-ai/Prompt-Engineering-Guide](https://github.com/dair-ai/Prompt-Engineering-Guide) — Open-source guide, papers, and resources for prompt engineering, RAG, and agents.
| Repo | ⭐ | Forks | Last commit | Activity | Priority | Difficulty | Summary |
|------|----|------|-------------|----------|----------|------------|---------|
| [dair-ai/Prompt-Engineering-Guide](https://github.com/dair-ai/Prompt-Engineering-Guide) | 76504 | 8386 | 2026-03-11 | Moderate | 5 | Beginner | Open-source guide, papers, and resources for prompt engineering, RAG, and agents. |
| [e2b-dev/awesome-ai-agents](https://github.com/e2b-dev/awesome-ai-agents) | 28751 | 3151 | 2026-07-09 | Active | 5 | Beginner | Curated collection of autonomous AI agents, frameworks, and resources. |
| [WooooDyy/LLM-Agent-Paper-List](https://github.com/WooooDyy/LLM-Agent-Paper-List) | 8166 | 493 | 2025-09-12 | Moderate | 5 | Intermediate | Paper list accompanying the SCIS survey on large-language-model-based agents. |
| [ai-boost/awesome-prompts](https://github.com/ai-boost/awesome-prompts) | 8473 | 801 | 2026-07-15 | Active | 4 | Beginner | Curated prompts and prompt-engineering resources from top GPTs. |
| [atfortes/Awesome-LLM-Reasoning](https://github.com/atfortes/Awesome-LLM-Reasoning) | 3650 | 212 | 2026-04-20 | Active | 4 | Intermediate | Curated papers on LLM reasoning from Chain-of-Thought to OpenAI o1 and DeepSeek-R1. |
| [DEEP-PolyU/Awesome-GraphRAG](https://github.com/DEEP-PolyU/Awesome-GraphRAG) | 2528 | 218 | 2026-06-02 | Active | 4 | Advanced | Curated surveys, papers, benchmarks, and projects on graph-based RAG. |
| [AGI-Edgerunners/LLM-Agents-Papers](https://github.com/AGI-Edgerunners/LLM-Agents-Papers) | 2330 | 152 | 2025-07-12 | Stale | 4 | Intermediate | Curated list of papers on LLM-based agents with daily auto-updates. |
| [hymie122/RAG-Survey](https://github.com/hymie122/RAG-Survey) | 1787 | 123 | 2024-08-20 | Stale | 4 | Intermediate | Curated papers on RAG foundations, enhancements, and AIGC applications. |
| [asinghcsu/AgenticRAG-Survey](https://github.com/asinghcsu/AgenticRAG-Survey) | 1700 | 181 | 2025-10-20 | Moderate | 4 | Advanced | Survey of agentic RAG systems enhanced with LLM agents. |
| [taichengguo/LLM_MultiAgents_Survey_Papers](https://github.com/taichengguo/LLM_MultiAgents_Survey_Papers) | 1291 | 65 | 2026-07-12 | Active | 4 | Advanced | Paper list for the IJCAI 2024 survey on LLM-based multi-agent systems. |
## 19. Awesome Lists
Top pick: [sindresorhus/awesome](https://github.com/sindresorhus/awesome) — The canonical Awesome meta-list of lists across all topics.
| Repo | ⭐ | Forks | Last commit | Activity | Priority | Difficulty | Summary |
|------|----|------|-------------|----------|----------|------------|---------|
| [sindresorhus/awesome](https://github.com/sindresorhus/awesome) | 485175 | 35874 | 2026-06-30 | Active | 5 | Beginner | The canonical Awesome meta-list of lists across all topics. |
| [f/awesome-chatgpt-prompts](https://github.com/f/awesome-chatgpt-prompts) | 165801 | 21442 | 2026-07-15 | Active | 5 | Beginner | Massive community-curated prompt collection for ChatGPT and other LLMs. |
| [dair-ai/Prompt-Engineering-Guide](https://github.com/dair-ai/Prompt-Engineering-Guide) | 76504 | 8386 | 2026-03-11 | Moderate | 5 | Intermediate | Comprehensive prompt engineering, RAG, and AI agent guide. |
| [openai/openai-cookbook](https://github.com/openai/openai-cookbook) | 74696 | 12644 | 2026-07-15 | Active | 5 | Intermediate | Official OpenAI cookbook with task-specific examples and recipes. |
| [e2b-dev/awesome-ai-agents](https://github.com/e2b-dev/awesome-ai-agents) | 28751 | 3151 | 2026-07-09 | Active | 5 | Intermediate | Curated list of AI autonomous agents, frameworks, and tools. |
| [Hannibal046/awesome-llm](https://github.com/Hannibal046/awesome-llm) | 27143 | 2634 | 2025-07-31 | Moderate | 5 | Intermediate | Curated list of Large Language Model resources and breakthroughs. |
| [awesome-selfhosted/awesome-selfhosted](https://github.com/awesome-selfhosted/awesome-selfhosted) | 305624 | 14320 | 2026-07-14 | Active | 4 | Intermediate | Curated list of self-hosted software and alternatives. |
| [mlabonne/llm-course](https://github.com/mlabonne/llm-course) | 80924 | 9425 | 2026-02-05 | Moderate | 4 | Advanced | Course to get into Large Language Models with roadmaps and notebooks. |
| [PlexPt/awesome-chatgpt-prompts-zh](https://github.com/PlexPt/awesome-chatgpt-prompts-zh) | 60972 | 13550 | 2026-04-28 | Active | 4 | Beginner | Chinese-language ChatGPT prompt engineering guide. |
| [ml-tooling/best-of-ml-python](https://github.com/ml-tooling/best-of-ml-python) | 23680 | 3142 | 2026-07-09 | Active | 4 | Advanced | Ranked list of awesome machine learning Python libraries, updated weekly. |
## Cross-cutting Comparison Tables
### Evaluation Layer comparison
| Repo | Stars | Sub-discipline | Prompts | Agents | Trajectory | CI/CD | License |
|------|------|----------------|:------:|:------:|:----------:|:-----:|---------|
| [langfuse/langfuse](https://github.com/langfuse/langfuse) | 31200 | All | Yes | Yes | Yes | NOASSERTION |
| [mlflow/mlflow](https://github.com/mlflow/mlflow) | 27040 | ? | ? | ? | ? | Apache-2.0 |
| [promptfoo/promptfoo](https://github.com/promptfoo/promptfoo) | 23285 | All | Yes | Yes | Yes | MIT |
| [comet-ml/opik](https://github.com/comet-ml/opik) | 20608 | All | Yes | Yes | Yes | Apache-2.0 |
| [openai/evals](https://github.com/openai/evals) | 18922 | All | Yes | Partial | Yes | NOASSERTION |
| [confident-ai/deepeval](https://github.com/confident-ai/deepeval) | 16868 | Agent | Yes | Yes | Yes | Apache-2.0 |
| [iterative/dvc](https://github.com/iterative/dvc) | 15750 | ? | ? | ? | ? | Apache-2.0 |
| [explodinggradients/ragas](https://github.com/explodinggradients/ragas) | 14850 | ? | ? | ? | ? | Apache-2.0 |
| [EleutherAI/lm-evaluation-harness](https://github.com/EleutherAI/lm-evaluation-harness) | 13289 | ? | ? | ? | ? | MIT |
| [arize-ai/phoenix](https://github.com/arize-ai/phoenix) | 10571 | ? | ? | ? | ? | NOASSERTION |
| [traceloop/openllmetry](https://github.com/traceloop/openllmetry) | 7302 | ? | ? | ? | ? | Apache-2.0 |
| [helicone/helicone](https://github.com/helicone/helicone) | 5948 | ? | ? | ? | ? | Apache-2.0 |
| [SWE-bench/SWE-bench](https://github.com/SWE-bench/SWE-bench) | 5427 | ? | ? | ? | ? | MIT |
| [agenta-ai/agenta](https://github.com/agenta-ai/agenta) | 4295 | ? | ? | ? | ? | NOASSERTION |
| [truera/trulens](https://github.com/truera/trulens) | 3438 | ? | ? | ? | ? | MIT |
| [hegelai/prompttools](https://github.com/hegelai/prompttools) | 3041 | ? | ? | ? | ? | Apache-2.0 |
| [UKGovernmentBEIS/inspect_ai](https://github.com/UKGovernmentBEIS/inspect_ai) | 2352 | Agent | Yes | Yes | Yes | MIT |
| [langchain-ai/langsmith-sdk](https://github.com/langchain-ai/langsmith-sdk) | 968 | ? | ? | ? | ? | MIT |
### Agent Frameworks comparison
| Repo | Stars | Multi-agent | Graph/state | Prod-ready | Language |
|------|------|:-----------:|:-----------:|:----------:|----------|
| [langflow-ai/langflow](https://github.com/langflow-ai/langflow) | 151902 | ? | ? | ? | ? |
| [langchain-ai/langchain](https://github.com/langchain-ai/langchain) | 141831 | ? | ? | ? | ? |
| [modelcontextprotocol/servers](https://github.com/modelcontextprotocol/servers) | 88505 | ? | ? | ? | ? |
| [All-Hands-AI/OpenHands](https://github.com/All-Hands-AI/OpenHands) | 80858 | Yes | Yes | Yes | Python |
| [microsoft/autogen](https://github.com/microsoft/autogen) | 59748 | Yes | Yes | Yes | Python |
| [crewAIInc/crewAI](https://github.com/crewAIInc/crewAI) | 55563 | Yes | Partial | Yes | Python |
| [run-llama/llama_index](https://github.com/run-llama/llama_index) | 50865 | ? | ? | ? | ? |
| [langchain-ai/langgraph](https://github.com/langchain-ai/langgraph) | 37349 | Yes | Yes | Yes | Python |
| [openai/openai-agents-python](https://github.com/openai/openai-agents-python) | 27926 | Yes | No | Yes | Python |
| [modelcontextprotocol/python-sdk](https://github.com/modelcontextprotocol/python-sdk) | 23617 | ? | ? | ? | ? |
### Coding Agents comparison
| Repo | Stars | Autonomous | Copilot | Self-host | Language |
|------|------|:---------:|:------:|:---------:|----------|
| [langchain-ai/langchain](https://github.com/langchain-ai/langchain) | 141832 | ? | ? | ? | ? |
| [anthropics/claude-code](https://github.com/anthropics/claude-code) | 137941 | ? | ? | ? | ? |
| [google-gemini/gemini-cli](https://github.com/google-gemini/gemini-cli) | 106001 | Yes | Yes | No | TypeScript |
| [openai/codex](https://github.com/openai/codex) | 98364 | Yes | Yes | No | Rust |
| [All-Hands-AI/OpenHands](https://github.com/All-Hands-AI/OpenHands) | 80859 | Yes | No | Yes | Python |
| [cline/cline](https://github.com/cline/cline) | 64683 | Yes | Yes | No | TypeScript |
| [block/goose](https://github.com/block/goose) | 51235 | Yes | Yes | Yes | Rust |
| [Aider-AI/aider](https://github.com/Aider-AI/aider) | 47400 | Yes | Yes | Yes | Python |
| [continuedev/continue](https://github.com/continuedev/continue) | 34889 | No | Yes | Yes | TypeScript |
| [TabbyML/tabby](https://github.com/TabbyML/tabby) | 33701 | ? | ? | ? | ? |
## Agent Engineering Architecture Stack
The Evaluation Layer is the emphasized central layer — it grades every other layer.
```mermaid
flowchart TB
subgraph INF[AI Infrastructure / AI 基础设施]
LiteLLM[vLLM · LiteLLM · Ollama]
end
subgraph FW[Frameworks / 框架]
LG[LangGraph · CrewAI · AutoGen]
end
subgraph HRN[Harness / Runtime 运行时]
TP[Temporal · LangGraph runtime]
end
subgraph SUR[Surfaces / 能力面]
MCP[MCP / Tools · Browser · Coding agents]
end
subgraph MEM[Memory / 记忆]
LT[Letta · Mem0 · Zep]
end
subgraph EV[★ Evaluation Layer 评估层 ★]
PF[promptfoo · OpenAI Evals · DeepEval · Inspect AI · Langfuse]
end
subgraph OBS[Observability / 可观测性]
OB[Phoenix · OpenLIT · Helicone]
end
subgraph BNC[Benchmarks / 基准]
SW[SWE-bench · WebArena · OSWorld · GAIA · τ-bench]
end
INF --> FW --> HRN --> SUR
MEM <--> HRN
EV <==> OBS
HRN -. traces 轨迹 .-> EV
EV -. scores 打分 .-> BNC
style EV fill:#fff4d6,stroke:#b8860b,stroke-width:3px,color:#000
```
## End-to-End Agent + Eval Flow
An explicit evaluation gate decides ship-vs-iterate on every change.
```mermaid
flowchart LR
U[User 用户] --> RT[Router / Gateway 路由网关]
RT --> AL[Agent Loop 智能体循环<br/>plan·act·observe]
AL --> TL[Tool / MCP 工具]
AL --> MM[Memory 记忆]
AL --> TR[Trace 轨迹]
TR --> EG{★ EVAL GATE 评估门控 ★<br/>assertions · judge · regression}
EG -->|pass 通过| SHP[✅ Ship 发布]
EG -->|fail / drift 失败/漂移| IT[🔁 Iterate prompt / agent 迭代]
IT --> AL
style EG fill:#fff4d6,stroke:#b8860b,stroke-width:3px,color:#000
```
## Recommended Learning Order
Begin with evaluation fundamentals *before* frameworks, to internalize that evals come first.
1. **promptfoo/promptfoo** — Write your first prompt regression suite + assertions _(Beginner/初级)_
2. **openai/evals** — Understand the canonical eval task + scoring harness shape _(Intermediate/中级)_
3. **confident-ai/deepeval** — Add LLM-judge metrics + multi-turn/trajectory eval _(Intermediate/中级)_
4. **UKGovernmentBEIS/inspect_ai** — Composable agent evals + scorer composition _(Advanced/高级)_
5. **langfuse/langfuse** — Trace + dataset + eval in one loop (EDD) _(Intermediate/中级)_
6. **langchain-ai/langgraph** — Build a stateful agent on a graph runtime _(Intermediate/中级)_
7. **All-Hands-AI/OpenHands** — See a full autonomous coding agent harness _(Advanced/高级)_
8. **modelcontextprotocol/servers** — Connect tools via MCP _(Beginner/初级)_
9. **browser-use/browser-use** — Drive the browser autonomously _(Intermediate/中级)_
10. **BerriAI/litellm** — Route + gatekeep model calls _(Beginner/初级)_
11. **letta-ai/letta** — Add long-term memory to an agent _(Advanced/高级)_
12. **EleutherAI/lm-evaluation-harness** — Run standard LLM benchmarks _(Intermediate/中级)_
13. **princeton-nlp/SWE-bench** — Measure a coding agent on real issues _(Advanced/高级)_
14. **stanfordnlp/dsp** — Programmatic prompt/agent optimization _(Advanced/高级)_
## Learning Roadmap by Role
Each role: the Evaluation Layer repo(s) they must learn first, then their stack.
### Backend / ML Engineer / 后端·ML 工程师
- **Eval first:** `promptfoo` → `DeepEval` → `Inspect AI`
- Then: `LangGraph` → `PydanticAI` → `LiteLLM` → `Mem0`
### Platform / SRE / 平台·SRE
- **Eval first:** `Langfuse` (trace + eval + EDD in CI) → `promptfoo` CLI gate
- Then: `LiteLLM` → `Phoenix`/`OpenLIT` → `Temporal` → `vLLM`/`SGLang`
### App / Product Builder / 应用·产品开发者
- **Eval first:** `promptfoo` (regression before each release) → `Opik`
- Then: `Mastra`/`LangGraph` → `MCP servers` → `Browser Use` → `Continue`/`Cline`
### Researcher / 研究者
- **Eval first:** `OpenAI Evals` → `Inspect AI` → `lm-evaluation-harness`
- Then: `SWE-bench`/`WebArena`/`GAIA`/`τ-bench` → `DSPy` → `HELM`
## Yearly Update — 2026
**What shifted this year:** reasoning models became the default substrate for hard agent tasks; **MCP standardized** tool attachment across frameworks and harnesses; **coding agents matured** from copilots into autonomous harnesses (OpenHands, Codex, Cline); and **agentic evals / EDD** crystallized into a recognized discipline — trajectory scoring and CI-for-AI moved from experiment to baseline.
### Maintenance checklist
- [ ] Re-run `python3 scripts/refresh_stats.py` quarterly to re-stamp stars/last-commit/activity
- [ ] Keep **≤ 10 repos per category** (tier-aware: Must Learn → Strongly Recommended → Emerging → Historical, by stars)
- [ ] Prune newly-archived repos, or demote them to the *Historical* tier
- [ ] **The Evaluation Layer is the most volatile** — review it most often; new eval harnesses appear fast
- [ ] Re-run `python3 scripts/gen_readme.py` after any `data/*.json` edit to regenerate this README
- [ ] Regenerate the `docs/` depth pages when a category's repo set changes materially
## Contributing
See [`CONTRIBUTING.md`](CONTRIBUTING.md). Short version: verify a repo with `gh api` before proposing it; never hand-edit stats in `data/*.json`; assign a ranking tier; keep ≤ 10 per category; regenerate README + the affected docs page.
## License
Code is MIT-licensed; the curated list content (README + docs) is CC-BY-4.0. See [`LICENSE`](LICENSE).
---
_Stats fetched via `gh api` on 2026-07-16 · ≤ 10 repos per category · generated by `scripts/gen_readme.py`._
Connection Info
You Might Also Like
No recommendations available