BM-25-Search

# bm-25 A lightweight BM25 search server for querying a folder of Markdown files, exposed as an [MCP](https://modelcontextprotocol.io) server. ## Repository layout ``` bm-25/ ├── mcp_server.py # MCP server exposing the `search` tool over stdio ├── search.py # BM25 indexing + search logic (also usable as a CLI) ├── requirements.txt # Python dependencies └── data/ # Place your .md files here ├── information_retrieval.md ├── machine_learning.md └── python_intro.md ``` ## Setup 1. **Install Python 3.9+** (if not already installed). 2. **Install dependencies:** ```bash pip install -r requirements.txt ``` ## Usage ### MCP server Run the server over stdio: ```bash python mcp_server.py ``` It registers a single tool, `search(query: str, top_n: int = 3)`, which returns the top-N BM25 matches (score, path, line range, and chunk text) from the markdown corpus in `data/`. OpenTelemetry spans are emitted to stderr so stdout stays clean for JSON-RPC. Example client config (e.g. `~/.config/claude/mcp.json` or equivalent): ```json { "mcpServers": { "bm25-search": { "command": "python", "args": ["/absolute/path/to/bm-25/mcp_server.py"] } } } ``` ### Docker Build the image: ```bash docker build -t bm25-mcp-server . ``` Configure Claude Desktop to use the containerized server (in `claude_desktop_config.json`): ```json { "mcpServers": { "bm25-search": { "command": "docker", "args": ["run", "-i", "--rm", "bm25-mcp-server"] } } } ``` ### CLI: interactive mode Run the script without arguments to enter an interactive query loop: ```bash python search.py ``` ``` Loading documents from '.../data' … Indexed 3 document(s): ['information_retrieval.md', 'machine_learning.md', 'python_intro.md'] Enter a search query (or 'quit' to exit): > machine learning algorithms 1. machine_learning.md (score: 0.4210) 2. python_intro.md (score: 0.1898) 3. information_retrieval.md (score: 0.0751) > quit ``` ### CLI: single-query mode Pass a query directly on the command line: ```bash python search.py "BM25 ranking" ``` ``` Loading documents from '.../data' … Indexed 3 document(s): ['information_retrieval.md', 'machine_learning.md', 'python_intro.md'] Query: 'BM25 ranking' 1. information_retrieval.md (score: 1.6854) ``` ## Adding your own documents Drop any `.md` files into the `data/` directory. They are automatically discovered and indexed the next time `search.py` is run. ## How it works 1. All `.md` files in `data/` are read and tokenized (lowercased, punctuation stripped). 2. A [BM25Okapi](https://github.com/dorianbrown/rank_bm25) index is built over the token lists. 3. Each query is tokenised the same way, and the top-scoring documents are returned. BM25 key parameters (can be tuned inside `search.py`): | Parameter | Default | Effect | |-----------|---------|--------| | `k1` | 1.5 | Term-frequency saturation | | `b` | 0.75 | Document-length normalization |