ai-agent-book

# Tool List This repository is the open-source main repository for the book "In-Depth Understanding of AI Agent: Design Principles and Engineering Practice", including **the full text** and **accompanying example code**. The full text, illustrations, and accompanying experimental code are all open-sourced, and we welcome you to run the experiments yourself, raise issues, and submit PRs. ## 📖 E-Book The full text and compiled PDF are located in the [`book/`](book/) directory: - **Text Source Code**: `book/introduction.md` (Introduction), `book/chapter1.md` ~ `book/chapter10.md` (Chapters 1-10), `book/afterword.md` (Afterword) - **Compiled PDF**: [`book/In-Depth-Understanding-AI-Agent-Li-Bojie-v1.0.pdf`](book/In-Depth-Understanding-AI-Agent-Li-Bojie-v1.0.pdf) - **Self-Compilation**: Install pandoc, xelatex, ElegantBook document class, and related fonts, then run ```bash cd book && bash build_pdf.sh ``` The diagrams are generated by `book/gen_*_figs.py`, stored in `book/images/`, and the layout details are in `book/preamble.tex` and `book/*.lua`. ## 📑 Content Overview (Chapters 1-10) The book revolves around the core formula **Agent = LLM + Context + Tools**, and the ten chapters are as follows: - **Chapter 1 · Agent Basics**: Starting from the new paradigm of "Model as Agent", establish the core formula of **Agent = LLM + Context + Tools**, and introduce Harness engineering—the engineering capabilities outside the model are the real competitive force. - **Chapter 2 · Context Engineering**: Context determines the upper limit of Agent's capabilities. In-depth exploration of large model API context structure, KV Cache friendly design, prompt engineering, dynamic prompt words and Agent Skills, status bar meta-information, and context compression strategies. - **Chapter 3 · User Memory and Knowledge Base**: Let Agent remember users across sessions and access external knowledge. Covers user memory system, RAG basic pipeline, and knowledge organization and retrieval beyond flat text (structured index, knowledge graph, etc.). - **Chapter 4 · Tools**: Tools are the hands of Agent. Discuss tool classification and general design principles, MCP protocol and tool selection challenges, perception/execution/collaboration three types of tools, and event-driven asynchronous Agent. - **Chapter 5 · Coding Agent and Code Generation**: Code is a "tool that creates new tools" and a meta-capability of a general Agent. Taking a production-level Coding Agent as an example, show the complete implementation of this strongest general tool. - **Chapter 6 · Evaluation of Agent**: Turn Agent's performance into comparable signals. From evaluation environment, dataset design, indicator system, to statistical significance, observability, evaluation-driven selection, and production-level internal evaluation and simulation environment. - **Chapter 7 · Model Post-Training**: A panoramic view of the three stages of pre-training, SFT, and RL. When to choose SFT, when to choose RL, RLHF, algorithm comparison, data and environment, and frontier exploration of letting the model learn tool calls and improve sample efficiency. - **Chapter 8 · Self-Evolution of Agent**: Grow without changing weights. Three learning paradigms, from learning from experience, active tool discovery, to "from tool user to tool creator", let Agent move from "smart" to "skilled". - **Chapter 9 · Multimodal and Real-Time Interaction**: Extend perception and action from text to voice, GUI, and physical world. Voice three paradigms (cascade/end-to-end full-modal/full-duplex), streaming voice perception and synthesis, Computer Use, and robot operation. - **Chapter 10 · Multi-Agent Collaboration**: The intelligence of a group can be higher than that of an individual. Multi-Agent classification framework, when to truly outperform a single Agent, sharing and non-sharing context collaboration, failure modes, and emerging "Agent society". ## 💻 Accompanying Code All projects are organized by **chapter**, corresponding to the ten chapters of the book, covering a complete learning path from basic concepts to advanced technologies. Each project is a complete and independently runnable example, with a directory of `chapterN/project name/`. ## 🚀 Chapter 1 · Agent Basics ### learning-from-experience - Reinforcement Learning vs. LLM Comparison `chapter1/learning-from-experience/` Compare traditional reinforcement learning (Q-learning) with LLM-based contextual learning, reproducing key insights from Shunyu Yao's "The Second Half" blog post. Showcase how LLM can surpass traditional RL with 250-400 times sample efficiency through a treasure hunt game. **Core Concepts**: Reinforcement Learning, Contextual Learning, Sample Efficiency, Prior Knowledge ### web-search-agent - Kimi K2 Model as Agent `chapter1/web-search-agent/` Implement an Agent with basic deep search capabilities, capable of multi-round searches and information integration. **Core Concepts**: Web Search, Model-Native Agent ### search-codegen - GPT-5 Native Tool Integration `chapter1/search-codegen/` Build an Agent with basic deep search capabilities and code sandbox capabilities, integrating network search, code execution, and other tools to achieve complex analysis. **Core Concepts**: Web Search, Code Generation, Model-Native Agent ### context - Context Ablation Study `chapter1/context/` Through a systematic ablation experiment, demonstrate the importance of each component of Agent context. Support multiple LLM providers (SiliconFlow Qwen, Byte Douobao, Moonlight Kimi), configure different context modes to observe Agent behavior changes. **Core Concepts**: Context Management, Tool Invocation, ReAct Loop, Ablation Study ## 🎯 Chapter 2 · Context Engineering ### local_llm_serving - Local LLM Deployment and Tool Invocation `chapter2/local_llm_serving/` Cross-platform local LLM deployment solution, automatically selecting the best backend (vLLM or Ollama). Showcase that even a 0.6B small model can achieve excellent tool invocation capabilities through good system design. Supports streaming response, displaying the thinking process in real-time. **Core Concepts**: Model Deployment, Chat Template, Streaming Processing, Tool Invocation ### attention_visualization - Attention Mechanism Visualization `chapter2/attention_visualization/` Visualize the complete input and output token sequence and attention weight distribution of LLM, deeply understanding how the model processes context and makes inferences and tool invocations. **Core Concepts**: Attention Mechanism, Token Analysis, Inference Process Visualization ### kv-cache - KV Cache Friendly Context Design `chapter2/kv-cache/` Explore the impact of different context management modes on KV Cache, demonstrating common error modes that can destroy cache efficiency. Through experiments, show how correct context design can significantly reduce latency and cost. **Core Concepts**: KV Cache, Context Optimization, Performance Tuning ### context-compression - Context Compression Strategy `chapter2/context-compression/` Implement and compare various context compression strategies, including summarization, key information extraction, semantic compression, etc. While maintaining Agent's capabilities, reduce token usage. **Core Concepts**: Context Compression, Token Optimization, Information Density ### prompt-engineering - Prompt Engineering Ablation Study `chapter2/prompt-engineering/` Extend the Tau-Bench framework, through systematic ablation experiments to quantify the impact of different prompt engineering factors on Agent performance. Showcase how tone style, instruction organization, tool description, and other factors affect task completion rate. **Core Concepts**: Prompt Engineering, Ablation Study, Performance Benchmarking ### system-hint - System Hint Optimization `chapter2/system-hint/` Study the impact of system hints on Agent behavior, exploring how to improve performance by optimizing system hints. **Core Concepts**: System Hint, Behavior Guidance, Hint Optimization ### log-sanitization - Log Desensitization `chapter2/log-sanitization/` Implement an intelligent log desensitization system, protecting sensitive data while retaining debugging information. **Core Concepts**: Privacy Protection, Log Processing, Data Security ## 📚 Chapter 3 · User Memory and Knowledge Base ### user-memory - User Memory System `chapter3/user-memory/` Build a long-term user memory system, enabling Agent to remember user preferences and historical interactions, providing personalized services. **Core Concepts**: Long-Term Memory, Personalization, User Modeling ### user-memory-evaluation - User Memory Evaluation Framework `chapter3/user-memory-evaluation/` Systematically evaluate the accuracy, relevance, and effectiveness of the user memory system, including multiple test scenarios and evaluation indicators. **Core Concepts**: Evaluation Framework, Test Cases, Performance Metrics ### dense-embedding - Dense Embedding Vector Retrieval Service `chapter3/dense-embedding/` Build a vector similarity search service, comparing and studying ANNOY (tree-based) and HNSW (graph-based) two approximate nearest neighbor index algorithms. Showcase the trade-offs between different index strategies in performance, memory usage, and update capabilities. **Core Concepts**: Dense Embedding, Vector Retrieval, ANN Algorithms, Semantic Search ### sparse-embedding - Sparse Indexing Engine `chapter3/sparse-embedding/` Implement a sparse vector search engine from scratch based on the BM25 algorithm, demonstrating the internal working mechanism of the search engine through rich logs and visualization interfaces, understanding word frequency weight calculation and inverted index principles. **Core Concepts**: Sparse Embedding, BM25, TF-IDF, Exact Matching ### retrieval-pipeline - Hybrid Retrieval Pipeline `chapter3/retrieval-pipeline/` Build a complete retrieval pipeline, combining dense retrieval, sparse retrieval, and neural re-ranking. Through carefully designed test cases, systematically demonstrate the complementary advantages of hybrid retrieval in different scenarios. **Core Concepts**: Hybrid Retrieval, Neural Re-Ranking, Cross-Encoder, Retrieval Fusion ### multimodal-agent - Multimodal Information Extraction `chapter3/multimodal-agent/` Compare three multimodal processing strategies: native multimodal processing, extraction as text, and tool-based analysis. Through ablation studies under a unified framework, reveal the trade-offs between different technical paths in fidelity, cost, and flexibility. **Core Concepts**: Multimodal, Visual Understanding, OCR, End-to-End Processing ### structured-index - Structured Index `chapter3/structured-index/` Implement and compare two advanced indexing strategies: RAPTOR (recursive abstract tree) and GraphRAG (knowledge graph). Through indexing technical manuals, demonstrate how to build a structured index that reflects the inherent hierarchy and association of knowledge. **Core Concepts**: RAPTOR, GraphRAG, Hierarchical Summarization, Knowledge Graph ### agentic-rag - Agentic RAG `chapter3/agentic-rag/` Compare the performance differences between traditional Non-Agentic RAG and Agentic RAG. Showcase how Agent can improve answer quality through the ReAct mode-driven iterative information retrieval when dealing with complex judicial QA. **Core Concepts**: Agentic RAG, ReAct Loop, Iterative Retrieval, Active Exploration ### agentic-rag-for-user-memory - Using Agentic RAG to Build User Memory `chapter3/agentic-rag-for-user-memory/` Apply the Agentic RAG framework to manage user dialogue history, processing cross-session memory retrieval through multi-round iterative search capabilities, achieving basic recall and multi-session retrieval capabilities. **Core Concepts**: User Memory, Dialogue History Indexing, Cross-Session Retrieval ### contextual-retrieval - Context-Aware Retrieval `chapter3/contextual-retrieval/` Implement the context-aware retrieval technology proposed by Anthropic, solving the context loss problem of traditional block methods by generating prefix summaries containing core context, reducing retrieval failure rate by 49-67%. **Core Concepts**: Context Enhancement, Prefix Generation, Semantic Anchoring, Retrieval Optimization ### contextual-retrieval-for-user-memory - Context-Aware User Memory System `chapter3/contextual-retrieval-for-user-memory/` Apply context-aware retrieval technology to user memory construction, combining Advanced JSON Cards and context-aware RAG to form a two-layer memory structure, achieving a higher level of active service capability. **Core Concepts**: Two-Layer Memory, Structured Facts, Contextual Retrieval, Active Service ### structured-knowledge-extraction - Structured Knowledge Extraction `chapter3/structured-knowledge-extraction/` Extract implicit knowledge from a large dataset of judicial cases, constructing a judgment experience model through factor analysis and importance modeling. Showcase how to convert implicit patterns in data into structured decision logic that can be used by Agent. **Core Concepts**: Knowledge Discovery, Factor Analysis, Data-Driven, Judgment Modeling ## 🛠️ Chapter 4 · Tools ### perception-tools - Perception Tool MCP Server `chapter4/perception-tools/` Build a comprehensive perception toolset, providing network search, multimodal understanding, file system operation, and public data source access capabilities. Most functions are based on free and open APIs (DuckDuckGo, Open-Meteo, Yahoo Finance, OpenStreetMap, etc.), no API keys required. **Core Concepts**: MCP Protocol, Multimodal Parsing, Public Data Sources, Document Understanding, Geographical Information Services ### execution-tools - Execution Tool MCP Server `chapter4/execution-tools/` Implement an execution toolset with safety mechanisms, including file operations, code interpreters, virtual terminals, and external system integration. Prevent dangerous operations through LLM secondary approval mechanisms, automatically summarize complex outputs, and verify code syntax. **Core Concepts**: MCP Protocol, Execution Safety, LLM Approval, Result Summarization, Automatic Verification ### collaboration-tools - Collaboration Tool MCP Server `chapter4/collaboration-tools/` Provide comprehensive collaboration capabilities, including browser automation (browser-use framework), human-machine collaboration (Human-in-the-Loop), multi-channel notifications (Email, Telegram, Slack, Discord), and timer management. Support administrator approval for sensitive operations and scheduled task scheduling. **Core Concepts**: MCP Protocol, Browser Automation, HITL Mode, Multi-Channel Notifications, Scheduled Tasks ### agent-with-event-trigger - Event-Triggered Agent and MCP Integration `chapter4/agent-with-event-trigger/` Modern event-driven Agent built on FastAPI, integrating all tools from the previous three MCP servers by default. Adopt a native asynchronous architecture to achieve clear MCP tool loading, receiving multi-source events (Web, instant messaging, GitHub, timer, etc.) through HTTP API. Provide automatic API documentation (Swagger UI) and backend monitoring capabilities. **Core Concepts**: FastAPI, Native Asynchronous, MCP Integration, Event-Driven, Automatic API Documentation, Tool Orchestration ### active-tool-selection - Active Tool Selection `chapter4/active-tool-selection/` Implement an intelligent tool selection mechanism, enabling Agent to actively select the most suitable tool combination based on task needs, rather than passively accepting a predefined toolset. **Core Concepts**: Tool Selection, Dynamic Tool Loading, Task Analysis ### async-agent - Asynchronous Agent Architecture `chapter4/async-agent/` Showcase how to build a truly asynchronous Agent, supporting concurrent task processing, non-blocking I/O, and efficient resource utilization. **Core Concepts**: Asynchronous Programming, Concurrent Processing, Event Loop, Non-Blocking I/O > In addition, `chapter4/docker-compose.yml` and `chapter4/DOCKER_DEPLOYMENT.md` provide a reference solution for containerized deployment of the above MCP tool servers. ## 💻 Chapter 5 · Coding Agent and Code Generation ### coding-agent - Production-Level Coding Agent `chapter5/coding-agent/` Production-level AI coding assistant built on Claude, adopting a pure Python implementation of all tools, no command-line dependencies required. Includes 17 complete tool implementations, covering file operations, search, shell operations, and project management. Specifically implemented a pure Python Grep tool, fully compatible with ripgrep functionality. **Core Features**: - Pure Python implementation, no command-line dependencies, especially suitable for Mac users - Complete tool suite: file read-write editing, pure Python regular search, directory listing, shell session management - System prompt technology: timestamp, tool invocation count, TODO list management, detailed error information - Persistent shell environment, automatic lint detection, streaming response support - Supports multiple LLM providers (Anthropic, OpenAI, OpenRouter) **Core Concepts**: Code Generation, File Editing, Pure Python Tools, System Prompts, Lint Detection, Multi-Provider Support ## 🎯 Chapter 6 · Evaluation of Agent ### terminal-bench - Terminal Environment Benchmark `chapter6/terminal-bench/` Terminal-Bench is a benchmark for testing AI Agent performance in real terminal environments. From compiling code to training models, setting up servers, evaluate Agent's ability to handle real end-to-end tasks. Includes a dataset and execution framework for approximately 100 tasks, supporting multiple Agent implementations. **Core Concepts**: Terminal Automation, Task Evaluation, Docker Sandbox, Benchmarking ### SWE-bench - Software Engineering Benchmark `chapter6/SWE-bench/` SWE-bench is a benchmark for evaluating large language models' ability to solve real GitHub issues. Given a code base and issue description, the model needs to generate a patch to solve the problem. Includes SWE-bench, SWE-bench Lite, SWE-bench Verified, and SWE-bench Multimodal. **Core Concepts**: Code Repair, GitHub Issues, Patch Generation, Docker Evaluation ### GAIA - General AI Assistant Benchmark `chapter6/GAIA/` GAIA aims to evaluate the next generation of LLMs (with tool enhancement, efficient prompts, search access, etc.). Includes 450+ non-trivial problems requiring different levels of tool usage and autonomy, with clear and unambiguous answers. Divided into 3 difficulty levels. **Core Concepts**: Tool Usage, Multi-Step Reasoning, Autonomy Evaluation ### OSWorld - Operating System-Level Agent Benchmark `chapter6/OSWorld/` Evaluate Agent's ability to execute complex tasks in a complete operating system environment, including file management, application operation, and system configuration. **Core Concepts**: Operating System Automation, Multi-Application Collaboration, System-Level Tasks ### android_world / android-world - Android Environment Benchmark `chapter6/android_world/` and `chapter6/android-world/` Evaluate Agent's performance in the Android mobile environment, including app navigation, UI interaction, and task completion capabilities. **Core Concepts**: Mobile Automation, Android UI, App Interaction ### tau2-bench - Tool-Enhanced Reasoning Benchmark `chapter6/tau2-bench/` Focus on evaluating Agent's ability to use tools for complex reasoning, including calculation, search, and data processing scenarios. **Core Concepts**: Tool-Enhanced Reasoning, Multi-Step Tasks, Tool Combination ### elo-leaderboard - ELO Leaderboard System `chapter6/elo-leaderboard/` Implement an Agent performance leaderboard based on the ELO scoring system, evaluating the relative capabilities of different Agents through match comparisons. **Core Concepts**: ELO Scoring, Relative Evaluation, Leaderboard System ## 🧠 Chapter 7 · Model Post-Training This chapter includes multiple model post-training projects, covering various techniques and application scenarios of supervised fine-tuning (SFT) and reinforcement learning (RL). ### AdaptThink - Adaptive Reasoning Depth `chapter7/AdaptThink/` and `chapter7/AdaptThink-original/` Let the reasoning model learn to adaptively choose the reasoning mode (Thinking vs NoThinking) based on the difficulty of the question. By constraint optimization and importance sampling, it significantly reduces the reasoning cost (45-69%) while improving accuracy. Based on the DeepSeek-R1-Distill-Qwen model, trained using the DAPO algorithm. **Core Concepts**: Adaptive Reasoning, Reasoning Cost Optimization, Constraint Optimization, Importance Sampling ### retool - Tool-Enhanced Mathematical Reasoning `chapter7/retool/` Improve the mathematical reasoning ability of large language models through multi-turn dialogue and code sandbox. Through SFT and RL two-stage training, let the model learn to use the code execution environment to assist in solving mathematical problems. Based on Qwen2.5-32B-Instruct, trained on the AIME 2024 dataset, using DAPO algorithm and SandboxFusion sandbox. **Core Concepts**: Tool Usage, Code Execution, Mathematical Reasoning, Multi-turn Dialogue, DAPO Algorithm ### AWorld / AWorld-train - Embodied Agent Training `chapter7/AWorld/` and `chapter7/AWorld-train/` Train embodied agents based on the AWorld framework, allowing agents to perform complex tasks in virtual environments and learn from experience. **Core Concepts**: Embodied Intelligence, Environment Interaction, Experience Learning ### SFTvsRL - SFT vs RL Comparative Study `chapter7/SFTvsRL/` Systematically compare the effects of Supervised Fine-Tuning (SFT) and Reinforcement Learning (RL) on different tasks, analyzing the pros and cons and applicable scenarios of the two methods. **Core Concepts**: SFT vs RL, Training Method Comparison, Performance Analysis ### verl - Efficient RL Training Framework `chapter7/verl/` verl is an efficient reinforcement learning framework designed for large language model RLHF training, supporting multiple algorithms such as PPO, GRPO, and DAPO. **Core Concepts**: RLHF, PPO, Distributed Training, Efficient Optimization ### Intuitor - Intuitive Reasoning Training `chapter7/Intuitor/` Train the model's intuitive reasoning ability, allowing it to make reasonable judgments quickly without needing a detailed thinking chain. **Core Concepts**: Intuitive Reasoning, Fast Decision-Making, Thinking Chain Optimization ### MultilingualReasoning - Multilingual Reasoning `chapter7/MultilingualReasoning/` Train the model's reasoning ability in multiple language environments, improving performance on cross-language tasks. **Core Concepts**: Multilingual, Cross-Lingual Reasoning, Language Generalization ### SpatialReasoning - Spatial Reasoning Training `chapter7/SpatialReasoning/` Focus on training the model's spatial reasoning ability, handling problems involving spatial relationships such as position, direction, and distance. **Core Concepts**: Spatial Reasoning, Geometric Understanding, Position Relationship ### SimpleVLA-RL - Visual-Language-Action RL `chapter7/SimpleVLA-RL/` Combine visual, language, and action reinforcement learning training, allowing the model to understand visual input and execute corresponding actions. **Core Concepts**: Visual-Language-Action, Multimodal RL, Embodied Intelligence ### continued-pretraining - Continued Pre-Training `chapter7/continued-pretraining/` Perform continued pre-training on specific domain data to improve the model's performance in the target domain. **Core Concepts**: Continued Pre-Training, Domain Adaptation, Knowledge Injection ### MiniMind-pretrain - Small Model Pre-Training `chapter7/MiniMind-pretrain/` Pre-train small language models from scratch, understanding the complete process and key technologies of pre-training. **Core Concepts**: Pre-Training, Small Models, Training Process ### sesame - Sequence Modeling and Evaluation `chapter7/sesame/` Focus on training and evaluation methods for sequence modeling tasks. **Core Concepts**: Sequence Modeling, Evaluation Methods, Performance Optimization ### orpheus - Music Generation and Understanding `chapter7/orpheus/` Train the model's music generation and understanding abilities. **Core Concepts**: Music Generation, Audio Understanding, Creative AI ### tinker-cookbook - Training Tips and Tricks `chapter7/tinker-cookbook/` Collect practical tips and best practices for model training. **Core Concepts**: Training Tips, Best Practices, Tuning Methods ## 🔄 Chapter 8 · Agent Self-Evolution This chapter focuses on letting agents continuously grow from experience without changing weights: precipitating successful trajectories into reusable experience, externalizing repetitive operations into tools, and distilling prompts and observations into models. ### gaia-experience - Learning from Successful Experience `chapter8/gaia-experience/` Implement a complete "learning-application" closed loop based on the AWorld framework and GAIA benchmark. Agents automatically summarize successful task trajectories into structured experience and retrieve and apply them in new tasks to achieve self-evolution. **Core Concepts**: Experience Learning, Strategy Summary, Trajectory Summary, Self-Evolution ### browser-use-rpa - Workflow Recording and Playback `chapter8/browser-use-rpa/` Implement a workflow recording system for browser automation, automatically encapsulating repetitive operation sequences into parameterized tools. By switching from expensive LLM inference to precise automated execution, achieve 3-5 times speed improvement. **Core Concepts**: Workflow Recording, RPA, Tool Generation, Externalized Learning ### prompt-distillation - Prompt Distillation `chapter8/prompt-distillation/` Distill the effects of complex prompts into model parameters, reducing prompt length during inference and solidifying contextual experience into parameterized knowledge. **Core Concepts**: Knowledge Distillation, Prompt Optimization, Parameterized Knowledge ### learn-from-observation - Observational Learning `chapter8/learn-from-observation/` Let models learn from observing human or other agent behavior without explicit labeling. **Core Concepts**: Observational Learning, Imitation Learning, Behavior Cloning ### feedback-guided-sampling - Feedback-Guided Sampling `chapter8/feedback-guided-sampling/` Use feedback signals to guide the model's sampling process, improving generation quality and task completion rate. **Core Concepts**: Feedback Learning, Sampling Optimization, Quality Control ## 🎙️ Chapter 9 · Multimodal and Real-Time Interaction ### live-audio - Real-Time Voice Conversation `chapter9/live-audio/` Real-time voice chat demonstration, integrating speech-to-text, AI conversation, and text-to-speech functions. Supports multiple AI service providers (OpenAI, OpenRouter, ARK, Siliconflow), providing low-latency conversation experience. **Core Features**: - Real-time voice input and VAD (Voice Activity Detection) - Multi-provider support: ASR (OpenAI Whisper, SenseVoice), LLM (GPT-4o, Gemini, Doubao), TTS (Fish Audio) - WebSocket real-time communication, low-latency audio streaming - Real-time latency monitoring and logging **Core Concepts**: Speech Recognition, Real-Time Conversation, TTS, WebSocket, Multi-Provider Architecture ### browser-use - Browser Automation Agent (Computer Use) `chapter9/browser-use/` Browser-Use is a powerful browser automation framework that allows LLMs to control browsers to complete complex tasks. Supports scenarios such as form filling, web navigation, and data extraction, and is a typical implementation of GUI automation (Computer Use). **Core Features**: - LLM-driven browser automation - Supports multiple LLMs (ChatBrowserUse, OpenAI, Google, local models) - Customizable tool extensions, authentication processing - Sandbox deployment support, cloud service integration **Core Concepts**: Browser Automation, Computer Use, Visual Understanding, Tool Extension ### claude-quickstarts - Claude Quick Start `chapter9/claude-quickstarts/` Quick start examples and best practices for Claude API, covering various usage scenarios. **Core Concepts**: Claude API, Prompt Engineering, Best Practices ## 🤝 Chapter 10 · Multi-Agent Collaboration ### use-computer-while-calling - Dual-Agent Architecture `chapter10/use-computer-while-calling/` Implement a dual-agent collaboration architecture of phone call agents and computer use agents. The two agents communicate directly through WebSocket without a coordinator. The phone agent handles voice interaction, and the computer agent performs browser automation, working in parallel to complete complex tasks that require voice and web operations. **Core Features**: - Direct agent-to-agent communication (no coordinator) - Standard tool calls for message passing - Parallel operations: voice conversation + browser automation - Simple JSON message protocol **Architecture Components**: - Phone Call Agent (Node.js): voice I/O, ASR/TTS, LLM conversation - Computer Use Agent (Python): browser automation, browser-use, web scraping - WebSocket communication: direct message passing between agents **Core Concepts**: Multi-Agent Collaboration, Agent-to-Agent Communication, Parallel Task Processing, Voice + Browser Integration ## 📖 Learning Suggestions ### Core Concept: Agent = Model + Context + Tools The core framework of this book is **Agent = Model + Context + Tools**, and these three components work together to achieve the intelligent behavior of agents: - **Model**: The brain of the agent, providing understanding, reasoning, and decision-making capabilities - **Context**: The operating system of the agent, including system instructions, conversation history, reasoning process, tool interaction records, etc. - **Tools**: The hands of the agent, allowing it to perceive the environment, execute operations, and interact with the external world ### Learning Path The learning path corresponds to each chapter of the book, unfolding around the three pillars: - **Chapter 1 · Basics**: Establish a complete cognitive framework for the agent system - understand the definition of agents in RL, compare traditional RL and LLM+RL paradigms, and master the core framework of **Agent = Model + Context + Tools**. **Key Insights**: The importance of prior knowledge surpasses algorithms and environments. - **Chapters 2-3 · Context**: Context is the operating system of the agent. Chapter 2 covers system prompts, KV Cache friendly design, context compression, and prompt engineering ablation; Chapter 3 covers user memory, dense/sparse/hybrid retrieval, Agentic RAG, context-aware retrieval, and structured knowledge extraction. **Key Insights**: A complete context includes system instructions, conversation history, reasoning process, tool interaction records, user memory, and external knowledge. - **Chapters 4-5 · Tools**: Tools are the bridge between agents and the world. Chapter 4 covers MCP tools for perception/execution/collaboration, event triggering, and asynchronous architecture; Chapter 5 delves into the complete implementation of production-level Coding Agents. **Key Insights**: Tool design should be universal (code interpreters are better than calculators), and code is the meta-capability that creates new tools. - **Chapters 6-7 · Model**: How to measure and amplify intelligence. Chapter 6 covers evaluation benchmarks such as Terminal-Bench, SWE-bench, GAIA, OSWorld, and Tau2-Bench; Chapter 7 covers SFT, RL, RLHF, sample efficiency, and post-training techniques. **Key Insights**: Independent verification signals are more reliable than "let the model think again"; the "model as agent" paradigm internalizes tool calls as native capabilities through RL. - **Chapter 8 · Self-Evolution**: Let agents grow from experience without changing weights - learning from experience, externalizing workflows into tools, and distilling prompts and observations into parameters. **Key Insights**: Learning from experience is the key for agents to move from "smart" to "skilled". - **Chapters 9-10 · Expansion and Collaboration**: Chapter 9 extends perception and action from text to voice, GUI, and physical world; Chapter 10 handles complex tasks through multi-agent collaboration. **Key Insights**: Each design decision in multi-agent systems can be found in the three elements of a single agent. ### Difficulty Level - **Beginner** (Chapters 1-2): Suitable for beginners, understanding basic concepts - **Intermediate** (Chapters 3-4): Requires some programming foundation, involving system integration - **Advanced** (Chapters 5-6): Requires strong programming ability, involving complex system design - **Expert** (Chapters 7-8): Requires deep learning and training/self-evolution experience - **Application** (Chapters 9-10): Comprehensive application of previous knowledge, building practical applications ### Practical Suggestions 1. **Hands-on Practice**: Each project is designed to be independently runnable, and it is recommended to personally run and modify the code 2. **Combine with the Book**: Cooperate with the corresponding chapters in the book repository [`book/`](book/) to read and understand the combination of theory and practice 3. **Experimental Comparison**: Multiple projects include ablation studies and comparative experiments, and deepen understanding through comparison 4. **Gradual Learning**: Start with simple projects and gradually move to complex systems 5. **Focus on Protocols**: The MCP server project in Chapter 4 demonstrates standardized tool protocols, which are key to building scalable agents ## 🔑 API Keys It is recommended to apply for API keys from several platforms for convenience: - **Kimi**: https://platform.moonshot.cn/ - **Siliconflow**: https://siliconflow.cn/ provides various open-source models, including DeepSeek, Qwen, etc. - **Volcano Engine**: https://www.volcengine.com/product/ark provides closed-source models (豆包) from ByteDance, with relatively low latency for domestic access - **OpenRouter**: https://openrouter.ai/ provides access to various closed-source and open-source models from abroad, including Gemini 2.5 Pro, Claude 4 Sonnet, OpenAI GPT-5, etc. (official API requires overseas IP and payment methods, OpenAI also requires overseas identity verification, and registration is more complicated) Model selection can refer to: https://01.me/2025/07/llm-api-setup/ ## 🤝 Contribution Welcome to contribute code improvements, bug fixes, or new example projects through Pull Requests. ## 📄 License The code of this project is for learning reference only, and specific license information can be found in each sub-project.