doctor

<div align="center"> <picture> <img alt="Doctor Logo" src="doctor.png" height="30%" width="30%"> </picture> <br> <h2>🩺 Doctor</h2> [](https://github.com/sisig-ai/doctor) [](LICENSE.md) [](https://github.com/sisig-ai/doctor/actions/workflows/pytest.yml) [](https://codecov.io/gh/sisig-ai/doctor) A tool for discovering, crawl, and indexing web sites to be exposed as an MCP server for LLM agents for better and more up-to-date reasoning and code generation. </div> --- ### 🔍 Overview Doctor provides a complete stack for: - Crawling web pages using crawl4ai - Chunking text with LangChain - Creating embeddings with OpenAI via litellm - Storing data in DuckDB with vector search support - Exposing search functionality via a FastAPI web service - Making these capabilities available to LLMs through an MCP server --- ### 🏗️ Core Infrastructure #### 🗄️ DuckDB - Database for storing document data and embeddings with vector search capabilities - Managed by unified Database class #### 📨 Redis - Message broker for asynchronous task processing #### 🕸️ Crawl Worker - Processes crawl jobs - Chunks text - Creates embeddings #### 🌐 Web Server - FastAPI service exposing endpoints - Fetching, searching, and viewing data - Exposing the MCP server --- ### 💻 Setup #### ⚙️ Prerequisites - Docker and Docker Compose - Python 3.10+ - uv (Python package manager) - OpenAI API key #### 📦 Installation 1. Clone this repository 2. Set up environment variables: ``` export OPENAI_API_KEY=your-openai-key ``` 3. Run the stack: ``` docker compose up ``` --- ### 👁 Usage 1. Go to http://localhost:9111/docs to see the OpenAPI docs 2. Look for the `/fetch_url` endpoint and start a crawl job by providing a URL 3. Use `/job_progress` to see the current job status 4. Configure your editor to use `http://localhost:9111/mcp` as an MCP server --- ### ☁️ Web API - `POST /fetch_url`: Start crawling a URL - `GET /search_docs`: Search indexed documents - `GET /job_progress`: Check crawl job progress - `GET /list_doc_pages`: List indexed pages - `GET /get_doc_page`: Get full text of a page --- ### 🔧 MCP Integration Ensure that your Docker Compose stack is up, and then add to your Cursor or VSCode MCP Servers configuration: ```json "doctor": { "type": "sse", "url": "http://localhost:9111/mcp" } ``` --- ### 🧪 Testing #### Running Tests To run all tests: ```bash # Run all tests with coverage report pytest ``` To run specific test categories: ```bash # Run only unit tests pytest -m unit # Run only async tests pytest -m async_test # Run tests for a specific component pytest tests/lib/test_crawler.py ``` #### Test Coverage The project is configured to generate coverage reports automatically: ```bash # Run tests with detailed coverage report pytest --cov=src --cov-report=term-missing ``` #### Test Structure - `tests/conftest.py`: Common fixtures for all tests - `tests/lib/`: Tests for library components - `test_crawler.py`: Tests for the crawler module - `test_chunker.py`: Tests for the chunker module - `test_embedder.py`: Tests for the embedder module - `test_database.py`: Tests for the unified Database class - `tests/common/`: Tests for common modules - `tests/services/`: Tests for service layer - `tests/api/`: Tests for API endpoints --- ### 🐞 Code Quality #### Pre-commit Hooks The project is configured with pre-commit hooks that run automatically before each commit: - `ruff check --fix`: Lints code and automatically fixes issues - `ruff format`: Formats code according to project style - Trailing whitespace removal - End-of-file fixing - YAML validation - Large file checks #### Setup Pre-commit To set up pre-commit hooks: ```bash # Install pre-commit uv pip install pre-commit # Install the git hooks pre-commit install ``` #### Running Pre-commit Manually You can run the pre-commit hooks manually on all files: ```bash # Run all pre-commit hooks pre-commit run --all-files ``` Or on staged files only: ```bash # Run on staged files pre-commit run ``` --- ### ⚖️ License This project is licensed under the MIT License - see the [LICENSE.md](LICENSE.md) file for details.