Content
# Local Vector Search for Jira, Confluence & Files (with MCP support)
- [Overview](#overview)
- [Updates](#updates)
- [How it works](#how-it-works)
- [Collection structure](#collection-structure)
- [Indexers configuration](#indexers-configuration)
- [Setup](#setup)
- [Env setup](#env-setup)
- [Local setup](#local-setup)
- [Docker setup](#docker-setup)
- [Authentication](#authentication)
- [Create collection for Confluence](#create-collection-for-confluence)
- [Create collection for Jira](#create-collection-for-jira)
- [Create collection for local files](#create-collection-for-local-files)
- [Update collection](#update-collection)
- [Search](#search)
- [Filtering by metafields](#filtering-by-metafields)
- [Fetch](#fetch)
- [Set up MCP](#set-up-mcp)
- [Unified MCP (recommended)](#unified-mcp-recommended)
- [Simple MCP](#simple-mcp)
- [Run unit tests](#run-unit-tests)
- [Good to know](#good-to-know)
## Overview
Index documents from Jira, Confluence, or local files into a local vector database and search them. All data stays on your machine.
**Key features:**
- Jira & Confluence (Server/Data Center and Cloud). Jira ticket = document, Confluence page = document
- Local files (.pdf, .pptx, .docx, etc.) via [Unstructured](https://github.com/Unstructured-IO/unstructured)
- **No data sent to third parties** (except when used as MCP with a remote AI agent)
- Hybrid search: vector search + BM25 keyword search, merged by Reciprocal Rank Fusion
- Incremental updates: no need to rebuild the full index each time
- Filter results by metafields (space, project, date, etc.)
- Ability to extend: add more data sources, search engines, embeddings, etc.
**Technologies:** [ChromaDB](https://github.com/chroma-core/chroma), [FAISS](https://github.com/facebookresearch/faiss), SQLite (BM25), [sentence-transformers](https://pypi.org/project/sentence-transformers/), [Unstructured](https://github.com/Unstructured-IO/unstructured), [LangChain](https://python.langchain.com/docs/introduction/)
More context: [Medium article](https://medium.com/@shnax0210/mcp-tool-for-vector-search-in-confluence-and-jira-6beeade658ba)
**Contacts:**
- Like it? Please star the repo
- Found a bug? [Open an issue](https://github.com/shnax0210/documents-vector-search/issues)
- Want to contribute? - feel free to do it via fork and sending a pull request
- Want to chat? [LinkedIn](https://www.linkedin.com/in/oleksii-shnepov-841447158/)
## Updates
- Check [UPDATES.md](UPDATES.md) for major updates.
- Some minor updates can be not added the file, so they can be found only in git history.
## How it works
```mermaid
flowchart TD
A[1. Create collection] -->|loads & indexes documents| B["Collection stored in ./data/collections/${name}"]
B --> C{What next?}
C --> D[2. Update collection]
C --> E[3. Search via CLI]
C --> F[4. Search via MCP]
D -->|indexes only new/changed docs| B
E -->|search| B
F -->|search| B
```
1. **Create** a collection — load and index documents from Jira, Confluence, or local files. Stored in `./data/collections/{name}`
2. **Update** — re-index only new or changed documents (much faster than full creation)
3. **Search** — find documents by text query via CLI
4. **MCP** — expose search as a tool for AI agents
### Collection structure
```mermaid
graph TD
A["./data/collections/${name}/"] --> B["documents/"]
A --> C["indexes/"]
A --> D["manifest.json"]
B --- B1["Loaded and converted documents"]
C --- C1["Vector and keyword index files"]
D --- D1["Collection metadata: name, last update time, reader config, index list"]
```
See `./main/core/documents_collection_creator.py` for creation/update details and `./main/core/documents_collection_searcher.py` for search details.
### Indexers configuration
When you create a collection, you can specify a list of `indexers` like: `--indexers "indexer_ChromaDb__embeddings_sentence-transformers_slash_all-MiniLM-L6-v2", "indexer_SqlLiteBM25"`. The indexers define what vector/keyword databases and embedding models are used. Database and embedding model are separated by `__`. For example:
- `indexer_ChromaDb__embeddings_sentence-transformers_slash_all-MiniLM-L6-v2` means that `ChromaDb` is used as vector database and [`sentence-transformers/all-MiniLM-L6-v2`](https://huggingface.co/sentence-transformers/all-MiniLM-L6-v2) is used as the embedding model. You can use any embedding model from next [list](https://huggingface.co/models?pipeline_tag=sentence-similarity&library=sentence-transformers&sort=trending), you only need to add prefix `embeddings_` and replace slash symbols with `_slash_`. For example, if you want to use ChromaDb with [BAAI/bge-m3](https://huggingface.co/BAAI/bge-m3) embedder model, indexer name should be: `indexer_ChromaDb__embeddings_BAAI_slash_bge-m3`;
- `indexer_SqlLiteBM25` means that SqlLite BM25 is used as search engine.
You can define as many indexers as you want, their search results will be combined by Reciprocal Rank Fusion.
## Setup
### Env setup
Either `uv` or `docker` can be used to run the project.
#### Local setup
1. Clone the repository
2. Install [uv](https://docs.astral.sh/uv/)
3. Run `uv sync` in the project root
#### Docker setup
A `Dockerfile` is included for running the tool without installing Python or uv locally.
**Build the image:**
```bash
docker build -t documents-vector-search .
```
**Run any command** by passing it as arguments and mounting a local `data/` folder:
```bash
docker run --rm \
-v $(pwd)/data:/app/data \
-e CONF_TOKEN="${yourToken}" \
documents-vector-search \
uv run confluence_collection_create_cmd_adapter.py \
--collection "confluence" \
--url "${baseConfluenceUrl}" \
--cql "${confluenceQuery}"
```
- Mount `-v $(pwd)/data:/app/data` so collections and caches are persisted on your host machine
- Pass credentials as `-e ENV_VAR=value` (see [Authentication](#authentication))
- The `data/` folder structure (`collections/`, `caches/`, `local_file_input/`) is created automatically inside the container
**Run the unified MCP HTTP server with Docker:**
```bash
docker run --rm \
-p 8000:8000 \
-v $(pwd)/data:/app/data \
documents-vector-search \
uv run collection_search_unified_mcp_adapter.py --http --port 8000
```
Then configure your MCP client to connect to `http://localhost:8000/mcp`.
### Authentication
Set environment variables before creating or updating Jira/Confluence collections (not needed for local files):
| Platform | Type | Environment Variables |
|---|---|---|
| Confluence Server/DC | Bearer token (recommended) | `CONF_TOKEN` |
| Confluence Server/DC | Login/Password | `CONF_LOGIN`, `CONF_PASSWORD` |
| Confluence Cloud | Email/API token | `ATLASSIAN_EMAIL`, `ATLASSIAN_TOKEN` ([get token](https://id.atlassian.com/manage/api-tokens)) |
| Jira Server/DC | Bearer token (recommended) | `JIRA_TOKEN` |
| Jira Server/DC | Login/Password | `JIRA_LOGIN`, `JIRA_PASSWORD` |
| Jira Cloud | Email/API token | `ATLASSIAN_EMAIL`, `ATLASSIAN_TOKEN` ([get token](https://id.atlassian.com/manage/api-tokens)) |
Cloud vs Server is auto-detected: URLs ending with `.atlassian.net` are treated as Cloud.
### Create collection for Confluence
```bash
uv run confluence_collection_create_cmd_adapter.py \
--collection "confluence" \
--url "${baseConfluenceUrl}" \
--cql "${confluenceQuery}"
```
- `--collection` — name of the collection (used later for update/search). Data stored in `./data/collections/{name}`
- `--url` — Confluence base URL (e.g., `https://confluence.example.com` or `https://your-domain.atlassian.net`)
- `--cql` — Confluence query, e.g., `"(space = 'MySpace') AND (lastModified >= '2025-01-01')"`
### Create collection for Jira
```bash
uv run jira_collection_create_cmd_adapter.py \
--collection "jira" \
--url "${baseJiraUrl}" \
--jql "${jiraQuery}"
```
- `--url` — Jira base URL (e.g., `https://jira.example.com` or `https://your-domain.atlassian.net`)
- `--jql` — Jira query, e.g., `"project = MyProject AND created >= -183d"`
### Create collection for local files
```bash
uv run files_collection_create_cmd_adapter.py --basePath "${pathToFolder}"
```
- Collection name defaults to the last folder name. Override with `--collection {name}`
- Unreadable files are skipped by default. Use `--failFast` to stop on first error
- Filter files with `--includePatterns "regex1" "regex2"` and `--excludePatterns "regex1" "regex2"`
- Uses [Unstructured](https://github.com/Unstructured-IO/unstructured) for parsing. Some formats may need [extra software](https://docs.unstructured.io/open-source/installation/full-installation#full-installation)
### Update collection
```bash
uv run collection_update_cmd_adapter.py --collection "${collectionName}"
```
### Search
```bash
uv run collection_search_cmd_adapter.py \
--collection "${collectionName}" \
--query "How to set up react project locally"
```
- `--includeMatchedChunksText` — include matched text chunks in results
- `--filter` — filter by metafields (see below)
- `--rrfK` — RRF constant for multi-index fusion (default: `60`)
#### Filtering by metafields
Works with ChromaDB and SQLite BM25 indexes.
**Syntax:**
```
field operator "value" and/or field operator "value"
```
Operators: `=`, `!=`, `>`, `>=`, `<`, `<=`. Use `and` / `or` to join conditions (mixing both is not supported).
**Confluence metafields:**
| Field | Description |
|---|---|
| `space` | Space key |
| `createdAt` | Page creation date |
| `createdBy` | Creator email (lowercase) |
| `lastModifiedAt` | Last update date |
**Examples:**
```bash
--filter 'space = "SPACE_KEY"'
--filter 'space = "SPACE_KEY" and lastModifiedAt > "2026-01-01"'
--filter '(space = "SPACE_KEY1" or space = "SPACE_KEY2") and lastModifiedAt > "2026-01-01"'
```
**Jira metafields:**
| Field | Description |
|---|---|
| `project` | Project key |
| `type` | Issue type (Bug, Task, Story, ...) |
| `status` | Status (Open, In Progress, Done, ...) |
| `priority` | Priority (High, Medium, Low, ...) |
| `epic` | Epic or parent issue key |
| `assignee` | Assignee email (lowercase) |
| `createdAt` | Issue creation date |
| `createdBy` | Creator email (lowercase) |
| `lastModifiedAt` | Last update date |
**Examples:**
```bash
--filter 'project = "PROJ"'
--filter 'project = "PROJ" and lastModifiedAt > "2026-01-01"'
--filter '(project = "PROJ1" or project = "PROJ2") and lastModifiedAt > "2026-01-01"'
```
**Files metafields:**
| Field | Description |
|---|---|
| `createdAt` | File creation date |
| `lastModifiedAt` | Last modified date |
| `folder1` | First subfolder of the file path |
| `folder2` | Second subfolder of the file path |
| `folderN` | Nth subfolder of the file path (only present if path has N or more subfolders) |
**Examples:**
```bash
--filter 'folder1 = "docs"'
--filter 'folder1 = "docs" and folder2 = "api"'
--filter '(folder2 = "api" or folder2 = "presentations") and lastModifiedAt > "2026-01-01"'
```
### Fetch
```bash
uv run collection_fetch_cmd_adapter.py \
--collection "${collectionName}" \
--id "${documentId}"
```
- `--id` — document ID to fetch (required)
- `--startLine` / `--endLine` — line range to return (default: 1–200)
- `--format` — output format: `json`, `json_with_indent` (default), or `toon`
### Set up MCP
There are two MCP server adapters:
| Adapter | Best for | Key differences |
|---|---|---|
| `collection_search_unified_mcp_adapter.py` | Modern AI models | All collections in one server. AI model chooses collection, filter and number of chunks. Supports stdio and HTTP transport. |
| `collection_search_mcp_stdio_adapter.py` | Simpler AI models or restricted setups | One collection per server. Collection, filter and other settings are hardcoded via CLI args. Stdio only. |
#### Unified MCP (recommended)
Add to your MCP config (e.g., `.vscode/mcp.json` for VS Code + GitHub Copilot):
```json
{
"servers": {
"dvs": {
"type": "stdio",
"command": "uv",
"args": [
"--directory", "${fullPathToRootProjectFolder}",
"run", "collection_search_unified_mcp_adapter.py"
]
}
}
}
```
- All collections from `./data/collections/` are available automatically
- Use `--collections "name1" "name2"` to limit which collections are exposed
- Use `--rrfK {number}` to tune Reciprocal Rank Fusion behavior for multi-index search
- Use `--defaultNumberOfChunks {number}` and `--maxNumberOfChunks {number}` to tune the number of text chunks returned by a single search
Or start as http server:
```bash
uv run collection_search_unified_mcp_adapter.py --http --port 8000
```
And setup mcp like:
```json
{
"servers": {
"dvs": {
"type": "http",
"url": "http://localhost:8000/mcp",
}
}
}
```
#### Simple MCP
```json
{
"servers": {
"search_${collectionName}": {
"type": "stdio",
"command": "uv",
"args": [
"--directory", "${fullPathToRootProjectFolder}",
"run", "collection_search_mcp_stdio_adapter.py",
"--collection", "${collectionName}"
]
}
}
}
```
- Replace `${collectionName}` and `${fullPathToRootProjectFolder}` with real values
- Use `--maxNumberOfChunks {number}` to control how many text chunks are returned (more = better search, but may exceed model context window)
- Use `--rrfK {number}` to tune Reciprocal Rank Fusion behavior for multi-index search
- Use `--filter` for metafield filtering ([details](#filtering-by-metafields))
**Prompt examples:**
- "Find info about AI use cases, search on Confluence, include all used links"
- "Find info about PDP carousel, search on Jira, include all used links"
### Run unit tests
If you develop the tool, you can run unit tests:
```
uv run pytest
```
## Good to know
- **Incremental updates** — only new/changed documents are re-indexed. Uses `lastModifiedDocumentTime` from `manifest.json` (5 mins for Jira and Confluence buffer to avoid missing concurrent updates);
- **Caching** — Jira/Confluence collection creation caches downloaded documents in `./data/caches/{hash}`. Same parameters = same cache. If you need fresh data, either run an update after creation, or delete the cache folder manually;
- there are more parameters in scripts, use "--help" to get more.
