doctor

<div align="center"> <picture> <img alt="Doctor Logo" src="doctor.png" height="30%" width="30%"> </picture> <br> <h2>🩺 Doctor</h2> [](https://github.com/sisig-ai/doctor) [](LICENSE.md) [](https://github.com/sisig-ai/doctor/actions/workflows/pytest.yml) [](https://codecov.io/gh/sisig-ai/doctor) A tool for discovering, crawl, and indexing web sites to be exposed as an MCP server for LLM agents for better and more up-to-date reasoning and code generation. </div> --- ### 🔍 Overview Doctor provides a complete stack for: - Crawling web pages using crawl4ai with hierarchy tracking - Chunking text with LangChain - Creating embeddings with OpenAI via litellm - Storing data in DuckDB with vector search support - Exposing search functionality via a FastAPI web service - Making these capabilities available to LLMs through an MCP server - Navigating crawled sites with hierarchical site maps --- ### 🏗️ Core Infrastructure #### 🗄️ DuckDB - Database for storing document data and embeddings with vector search capabilities - Managed by unified Database class #### 📨 Redis - Message broker for asynchronous task processing #### 🕸️ Crawl Worker - Processes crawl jobs - Chunks text - Creates embeddings #### 🌐 Web Server - FastAPI service exposing endpoints - Fetching, searching, and viewing data - Exposing the MCP server --- ### 💻 Setup #### ⚙️ Prerequisites - Docker and Docker Compose - Python 3.10+ - uv (Python package manager) - OpenAI API key #### 📦 Installation 1. Clone this repository 2. Set up environment variables: ``` export OPENAI_API_KEY=your-openai-key ``` 3. Run the stack: ``` docker compose up ``` --- ### 👁 Usage 1. Go to http://localhost:9111/docs to see the OpenAPI docs 2. Look for the `/fetch_url` endpoint and start a crawl job by providing a URL 3. Use `/job_progress` to see the current job status 4. Configure your editor to use `http://localhost:9111/mcp` as an MCP server --- ### ☁️ Web API #### Core Endpoints - `POST /fetch_url`: Start crawling a URL - `GET /search_docs`: Search indexed documents - `GET /job_progress`: Check crawl job progress - `GET /list_doc_pages`: List indexed pages - `GET /get_doc_page`: Get full text of a page #### Site Map Feature The Maps feature provides a hierarchical view of crawled websites, making it easy to navigate and explore the structure of indexed sites. **Endpoints:** - `GET /map`: View an index of all crawled sites - `GET /map/site/{root_page_id}`: View the hierarchical tree structure of a specific site - `GET /map/page/{page_id}`: View a specific page with navigation (parent, siblings, children) - `GET /map/page/{page_id}/raw`: Get the raw markdown content of a page **Features:** - **Hierarchical Navigation**: Pages maintain parent-child relationships, allowing you to navigate through the site structure - **Domain Grouping**: Pages from the same domain crawled individually are automatically grouped together - **Automatic Title Extraction**: Page titles are extracted from HTML or markdown content - **Breadcrumb Navigation**: Easy navigation with breadcrumbs showing the path from root to current page - **Sibling Navigation**: Quick access to pages at the same level in the hierarchy - **Legacy Page Support**: Pages crawled before hierarchy tracking are grouped by domain for easy access - **No JavaScript Required**: All navigation works with pure HTML and CSS for maximum compatibility **Usage Example:** 1. Crawl a website using the `/fetch_url` endpoint 2. Visit `/map` to see all crawled sites 3. Click on a site to view its hierarchical structure 4. Navigate through pages using the provided links --- ### 🔧 MCP Integration Ensure that your Docker Compose stack is up, and then add to your Cursor or VSCode MCP Servers configuration: ```json "doctor": { "type": "sse", "url": "http://localhost:9111/mcp" } ``` --- ### 🧪 Testing #### Running Tests To run all tests: ```bash # Run all tests with coverage report pytest ``` To run specific test categories: ```bash # Run only unit tests pytest -m unit # Run only async tests pytest -m async_test # Run tests for a specific component pytest tests/lib/test_crawler.py ``` #### Test Coverage The project is configured to generate coverage reports automatically: ```bash # Run tests with detailed coverage report pytest --cov=src --cov-report=term-missing ``` #### Test Structure - `tests/conftest.py`: Common fixtures for all tests - `tests/lib/`: Tests for library components - `test_crawler.py`: Tests for the crawler module - `test_crawler_enhanced.py`: Tests for enhanced crawler with hierarchy tracking - `test_chunker.py`: Tests for the chunker module - `test_embedder.py`: Tests for the embedder module - `test_database.py`: Tests for the unified Database class - `test_database_hierarchy.py`: Tests for database hierarchy operations - `tests/common/`: Tests for common modules - `tests/services/`: Tests for service layer - `test_map_service.py`: Tests for the map service - `tests/api/`: Tests for API endpoints - `test_map_api.py`: Tests for map API endpoints - `tests/integration/`: Integration tests - `test_processor_enhanced.py`: Tests for enhanced processor with hierarchy --- ### 🐞 Code Quality #### Pre-commit Hooks The project is configured with pre-commit hooks that run automatically before each commit: - `ruff check --fix`: Lints code and automatically fixes issues - `ruff format`: Formats code according to project style - Trailing whitespace removal - End-of-file fixing - YAML validation - Large file checks #### Setup Pre-commit To set up pre-commit hooks: ```bash # Install pre-commit uv pip install pre-commit # Install the git hooks pre-commit install ``` #### Running Pre-commit Manually You can run the pre-commit hooks manually on all files: ```bash # Run all pre-commit hooks pre-commit run --all-files ``` Or on staged files only: ```bash # Run on staged files pre-commit run ``` --- ### ⚖️ License This project is licensed under the MIT License - see the [LICENSE.md](LICENSE.md) file for details.