Content
<div align="center">
<picture>
<img alt="Doctor Logo" src="doctor.png" height="30%" width="30%">
</picture>
<br>
<h2>🩺 Doctor</h2>
[](https://github.com/sisig-ai/doctor)
[](LICENSE.md)
[](https://github.com/sisig-ai/doctor/actions/workflows/pytest.yml)
[](https://codecov.io/gh/sisig-ai/doctor)
A tool for discovering, crawl, and indexing web sites to be exposed as an MCP server for LLM agents for better and more up-to-date reasoning and code generation.
</div>
---
### 🔍 Overview
Doctor provides a complete stack for:
- Crawling web pages using crawl4ai with hierarchy tracking
- Chunking text with LangChain
- Creating embeddings with OpenAI via litellm
- Storing data in DuckDB with vector search support
- Exposing search functionality via a FastAPI web service
- Making these capabilities available to LLMs through an MCP server
- Navigating crawled sites with hierarchical site maps
---
### 🏗️ Core Infrastructure
#### 🗄️ DuckDB
- Database for storing document data and embeddings with vector search capabilities
- Managed by unified Database class
#### 📨 Redis
- Message broker for asynchronous task processing
#### 🕸️ Crawl Worker
- Processes crawl jobs
- Chunks text
- Creates embeddings
#### 🌐 Web Server
- FastAPI service exposing endpoints
- Fetching, searching, and viewing data
- Exposing the MCP server
---
### 💻 Setup
#### ⚙️ Prerequisites
- Docker and Docker Compose
- Python 3.10+
- uv (Python package manager)
- OpenAI API key
#### 📦 Installation
1. Clone this repository
2. Set up environment variables:
```
export OPENAI_API_KEY=your-openai-key
```
3. Run the stack:
```
docker compose up
```
---
### 👁 Usage
1. Go to http://localhost:9111/docs to see the OpenAPI docs
2. Look for the `/fetch_url` endpoint and start a crawl job by providing a URL
3. Use `/job_progress` to see the current job status
4. Configure your editor to use `http://localhost:9111/mcp` as an MCP server
---
### ☁️ Web API
#### Core Endpoints
- `POST /fetch_url`: Start crawling a URL
- `GET /search_docs`: Search indexed documents
- `GET /job_progress`: Check crawl job progress
- `GET /list_doc_pages`: List indexed pages
- `GET /get_doc_page`: Get full text of a page
#### Site Map Feature
The Maps feature provides a hierarchical view of crawled websites, making it easy to navigate and explore the structure of indexed sites.
**Endpoints:**
- `GET /map`: View an index of all crawled sites
- `GET /map/site/{root_page_id}`: View the hierarchical tree structure of a specific site
- `GET /map/page/{page_id}`: View a specific page with navigation (parent, siblings, children)
- `GET /map/page/{page_id}/raw`: Get the raw markdown content of a page
**Features:**
- **Hierarchical Navigation**: Pages maintain parent-child relationships, allowing you to navigate through the site structure
- **Domain Grouping**: Pages from the same domain crawled individually are automatically grouped together
- **Automatic Title Extraction**: Page titles are extracted from HTML or markdown content
- **Breadcrumb Navigation**: Easy navigation with breadcrumbs showing the path from root to current page
- **Sibling Navigation**: Quick access to pages at the same level in the hierarchy
- **Legacy Page Support**: Pages crawled before hierarchy tracking are grouped by domain for easy access
- **No JavaScript Required**: All navigation works with pure HTML and CSS for maximum compatibility
**Usage Example:**
1. Crawl a website using the `/fetch_url` endpoint
2. Visit `/map` to see all crawled sites
3. Click on a site to view its hierarchical structure
4. Navigate through pages using the provided links
---
### 🔧 MCP Integration
Ensure that your Docker Compose stack is up, and then add to your Cursor or VSCode MCP Servers configuration:
```json
"doctor": {
"type": "sse",
"url": "http://localhost:9111/mcp"
}
```
---
### 🧪 Testing
#### Running Tests
To run all tests:
```bash
# Run all tests with coverage report
pytest
```
To run specific test categories:
```bash
# Run only unit tests
pytest -m unit
# Run only async tests
pytest -m async_test
# Run tests for a specific component
pytest tests/lib/test_crawler.py
```
#### Test Coverage
The project is configured to generate coverage reports automatically:
```bash
# Run tests with detailed coverage report
pytest --cov=src --cov-report=term-missing
```
#### Test Structure
- `tests/conftest.py`: Common fixtures for all tests
- `tests/lib/`: Tests for library components
- `test_crawler.py`: Tests for the crawler module
- `test_crawler_enhanced.py`: Tests for enhanced crawler with hierarchy tracking
- `test_chunker.py`: Tests for the chunker module
- `test_embedder.py`: Tests for the embedder module
- `test_database.py`: Tests for the unified Database class
- `test_database_hierarchy.py`: Tests for database hierarchy operations
- `tests/common/`: Tests for common modules
- `tests/services/`: Tests for service layer
- `test_map_service.py`: Tests for the map service
- `tests/api/`: Tests for API endpoints
- `test_map_api.py`: Tests for map API endpoints
- `tests/integration/`: Integration tests
- `test_processor_enhanced.py`: Tests for enhanced processor with hierarchy
---
### 🐞 Code Quality
#### Pre-commit Hooks
The project is configured with pre-commit hooks that run automatically before each commit:
- `ruff check --fix`: Lints code and automatically fixes issues
- `ruff format`: Formats code according to project style
- Trailing whitespace removal
- End-of-file fixing
- YAML validation
- Large file checks
#### Setup Pre-commit
To set up pre-commit hooks:
```bash
# Install pre-commit
uv pip install pre-commit
# Install the git hooks
pre-commit install
```
#### Running Pre-commit Manually
You can run the pre-commit hooks manually on all files:
```bash
# Run all pre-commit hooks
pre-commit run --all-files
```
Or on staged files only:
```bash
# Run on staged files
pre-commit run
```
---
### ⚖️ License
This project is licensed under the MIT License - see the [LICENSE.md](LICENSE.md) file for details.
Connection Info
You Might Also Like
MarkItDown MCP
Converting files and office documents to Markdown.
Time
Obtaining current time information and converting time between different...
Filesystem
Model Context Protocol Servers
Sequential Thinking
Offers a structured approach to dynamic and reflective problem-solving,...
Git
Model Context Protocol Servers
Context 7
Context7 MCP Server -- Up-to-date code documentation for LLMs and AI code editors