Content
> [!IMPORTANT]
> **This repository has been retired.** It is no longer maintained.
>
> This package is now part of the **[AppliedIR/sift-mcp](https://github.com/AppliedIR/sift-mcp)** monorepo.
>
> Documentation: [appliedir.github.io/aiir](https://appliedir.github.io/aiir/)
---
# RAG-MCP: Incident Response Knowledge Server
MCP server providing semantic search over incident response knowledge from 23 authoritative security sources.
## Installation Options
### Option A: As Part of Claude-IR (Recommended)
This MCP is designed as a component of the [Claude-IR](https://github.com/scriptedstatement/claude-ir) AI-assisted incident response workstation.
```bash
git clone https://github.com/scriptedstatement/claude-ir.git
cd claude-ir
./setup.sh
claude
```
**Benefits of Claude-IR installation:**
- Guided setup with component selection
- Pre-configured MCP integration
- Works alongside windows-triage-mcp (file validation) and opencti-mcp (threat intel)
- Forensic discipline rules and investigation workflows
### Option B: Standalone Installation
Use standalone when you only need RAG search without the full IR workstation.
```bash
git clone https://github.com/scriptedstatement/forensic-rag-mcp.git
cd forensic-rag-mcp
# Create virtual environment
python3 -m venv .venv
source .venv/bin/activate
# Install dependencies
pip install -e .
# Build the index (fetches online sources, ~5 minutes first time)
python -m rag_mcp.build
# Check status
python -m rag_mcp.status
# Run MCP server (for Claude Code integration)
python -m rag_mcp.server
```
**For detailed setup options (online sources vs. custom content):** See `SETUP.md`
## Features
- **23 Online Sources**: Auto-synced from authoritative security repositories
- **Semantic Search**: BGE embeddings with ChromaDB vector store
- **MCP Integration**: Tools for Claude Code (search, list_sources, get_stats)
- **User Documents**: Add your own knowledge via watched folder or one-time ingest
- **Incremental Updates**: Fast refresh without full rebuild
- **Filesystem Safety**: Sentinel-based deletion guards prevent accidental data loss
- **Network Hardening**: HTTPS-only, size limits, retry with backoff, SSRF protection
- **Exact Source Filtering**: Deterministic `source_ids` filter for reliable results
## Online Sources
| Source | Description | Records |
|--------|-------------|---------|
| sigma | SigmaHQ Detection Rules | ~3,100 |
| mitre_attack | MITRE ATT&CK (techniques, groups, malware, campaigns, mitigations) | ~2,100 |
| atomic | Atomic Red Team Tests | ~1,800 |
| elastic | Elastic Detection Rules | ~1,500 |
| cisa_kev | CISA Known Exploited Vulnerabilities | ~1,500 |
| capec | MITRE CAPEC Attack Patterns | ~1,500 |
| splunk_security | Splunk Security Content | ~1,000 |
| kape | KAPE Targets & Modules | ~800 |
| forensic_artifacts | ForensicArtifacts Definitions | ~700 |
| hijacklibs | DLL Hijacking Database | ~600 |
| loldrivers | LOLDrivers Vulnerable Driver Database | ~500 |
| mitre_d3fend | MITRE D3FEND Countermeasures | ~490 |
| gtfobins | GTFOBins (Linux) | ~450 |
| velociraptor | Velociraptor Artifacts | ~300 |
| lolbas | LOLBAS Project | ~130 |
| mitre_car | MITRE CAR Analytics | ~100 |
| stratus_red_team | Cloud Attack Techniques | ~80 |
| mbc | MITRE MBC Malware Behavior Catalog | ~650 |
| mitre_atlas | MITRE ATLAS AI/ML Attack Framework | ~50 |
| mitre_engage | MITRE Engage Adversary Engagement | ~45 |
| chainsaw | Chainsaw Forensic Detection Rules | ~110 |
| hayabusa | Hayabusa Built-in Detection Rules | ~190 |
| forensic_clarifications | Authoritative Forensic Artifact Clarifications | 5 |
## Commands
### Build Index
```bash
# Full build (fetch all sources)
python -m rag_mcp.build
# Use cached sources only (offline mode)
python -m rag_mcp.build --skip-online
# Force re-fetch all sources
python -m rag_mcp.build --force-fetch
# Preview what would be built
python -m rag_mcp.build --dry-run
```
### Refresh Index
```bash
# Check for updates and apply
python -m rag_mcp.refresh
# Check only (don't apply)
python -m rag_mcp.refresh --check-only
# Refresh specific source
python -m rag_mcp.refresh --source sigma
```
### Check Status
```bash
# Show status
python -m rag_mcp.status
# JSON output
python -m rag_mcp.status --json
# Skip update checks (faster)
python -m rag_mcp.status --no-check
```
## Bundled Reference Content
The `knowledge/` folder ships with JSONL extractions from two reference sources, included with permission:
| Source | Original Location |
|--------|-------------------|
| **SANS Posters** | https://www.sans.org/posters |
| **Applied Incident Response** | https://www.appliedincidentresponse.com/resources |
These are indexed automatically during `rag_mcp.build`. To build without bundled content, use `--no-bundled`.
## User Documents
### Watched Documents (knowledge/ folder)
Place files in `knowledge/` - they're automatically indexed on build/refresh:
```
knowledge/
├── my-playbook.md # Markdown
├── ioc-list.txt # Plain text
└── detections.jsonl # JSONL records
```
Supported formats: `.txt`, `.md`, `.json`, `.jsonl`
### One-Time Ingestion
For sensitive documents you don't want to keep on disk:
```bash
# Ingest with friendly name
python -m rag_mcp.ingest /path/to/doc.txt --name "my-doc"
# List ingested documents
python -m rag_mcp.ingest --list
# Remove specific document
python -m rag_mcp.ingest --remove "my-doc"
# Remove all ingested
python -m rag_mcp.ingest --remove-all
```
## MCP Tools
When running as MCP server, exposes these tools:
| Tool | Description |
|------|-------------|
| `search` | Semantic search with optional filters |
| `list_sources` | List available knowledge sources |
| `get_stats` | Get index statistics |
### Search Filters
```python
# Filter by source
search(query="credential dumping", source="sigma")
# Filter by MITRE technique
search(query="persistence", technique="T1053")
# Filter by platform
search(query="privilege escalation", platform="windows")
```
## MCP Configuration
Add to your MCP configuration (e.g., `.claude/mcp.json`):
```json
{
"mcpServers": {
"rag-knowledge": {
"command": "/path/to/forensic-rag-mcp/.venv/bin/python",
"args": ["-m", "rag_mcp.server"],
"cwd": "/path/to/forensic-rag-mcp",
"env": {
"PYTHONPATH": "/path/to/forensic-rag-mcp/src"
}
}
}
}
```
## Configuration
Settings are loaded via `Config` dataclass (`config.py`) with an explicit `validate()` method and global singleton caching in `get_config()`. Set `RAG_LOG_FORMAT=json` for structured JSON logging.
| Variable | Default | Description |
|----------|---------|-------------|
| `RAG_INDEX_DIR` | `./data` | ChromaDB location |
| `RAG_KNOWLEDGE_DIR` | `./knowledge` | User documents |
| `RAG_MODEL_NAME` | `BAAI/bge-base-en-v1.5` | Embedding model |
| `GITHUB_TOKEN` | (optional) | Higher API rate limits (see Security section) |
| `RAG_MAX_DOWNLOAD_BYTES` | `62914560` (60MB) | Maximum download size for network fetches |
| `RAG_ALLOW_HTTP` | `false` | Allow HTTP (not recommended; HTTPS enforced by default) |
| `RAG_FETCH_MAX_RETRIES` | `3` | Max retries for transient network failures |
| `RAG_MAX_TOP_K` | `50` | Maximum top_k for queries |
| `RAG_MAX_QUERY_LENGTH` | `1000` | Maximum query string length |
| `RAG_LOG_FORMAT` | `text` | Log format: "text" or "json" |
## Project Structure
```
rag-mcp/
├── src/rag_mcp/
│ ├── server.py # MCP server
│ ├── index.py # Search engine wrapper
│ ├── sources.py # Online source management (23 sources)
│ ├── ingest.py # User document processing
│ ├── build.py # Full index builder
│ ├── refresh.py # Incremental updates
│ ├── status.py # Status reporting
│ ├── config.py # Centralized configuration management
│ ├── fs_safety.py # Filesystem safety guardrails
│ ├── constants.py # Project-wide constants
│ └── utils.py # Shared utilities
├── data/ # Built index (gitignored, users rebuild)
│ ├── chroma/ # ChromaDB vector store
│ ├── sources/ # Cached online source JSONL
│ └── *.json # State files
├── knowledge/ # User documents - .txt, .md, .json, .jsonl (watched folder)
└── tests/ # Test suite
```
## Score Interpretation
| Score | Quality | Action |
|-------|---------|--------|
| 0.85+ | Excellent | High confidence match |
| 0.75-0.84 | Good | Relevant result |
| 0.65-0.74 | Fair | May be tangential |
| < 0.65 | Weak | Likely not relevant |
## Query Analysis and Tuning
The system includes tools for monitoring query quality and tuning thresholds based on actual usage patterns.
### Logging Setup
**Query logging is OFF by default** for privacy. Query text is logged verbatim, which may include sensitive search terms. Only enable logging if you understand the privacy implications.
**Privacy Considerations:**
- Query logs contain the exact text users search for
- In regulated environments (HIPAA, SOX, etc.), consider encrypting log files at rest
- Ensure log file permissions restrict access appropriately
To enable query logging, you must explicitly add handlers:
```python
# In your logging configuration
import logging
# Create handlers FIRST
metrics_handler = logging.FileHandler("logs/query_metrics.log")
attention_handler = logging.FileHandler("logs/attention.log")
# Get loggers and add handlers (this enables logging)
metrics_logger = logging.getLogger("rag_mcp.query_metrics")
metrics_logger.setLevel(logging.INFO)
metrics_logger.addHandler(metrics_handler)
metrics_logger.propagate = True # Enable propagation if you want console output too
attention_logger = logging.getLogger("rag_mcp.attention")
attention_logger.setLevel(logging.WARNING)
attention_logger.addHandler(attention_handler)
attention_logger.propagate = True
```
### Analyze and Tune
Run the analysis tool periodically to review query performance and adjust thresholds:
```bash
# Full interactive workflow (analyze, recommend, approve)
python -m rag_mcp.analyze_queries
# Just view report without making changes
python -m rag_mcp.analyze_queries --report-only
# Analyze last 7 days only
python -m rag_mcp.analyze_queries --since 7d
# Set approver name for audit trail
python -m rag_mcp.analyze_queries --approver "your-name"
# Export analysis to JSON
python -m rag_mcp.analyze_queries --export analysis.json
```
### Interactive Workflow
The tool is designed for both human operators and AI agents:
1. **Analysis**: Reads query logs and computes statistics
2. **Report**: Shows query type breakdown, attention issues, content gaps
3. **Recommendations**: Generates specific tuning suggestions with rationale
4. **Approval**: Prompts for Y/N approval on each recommendation
5. **Apply**: Saves approved changes to `data/tuning_config.json`
6. **Audit Trail**: Records who approved each change and why
Example session:
```
$ python -m rag_mcp.analyze_queries
Analyzing query logs...
======================================================================
RAG QUERY ANALYSIS REPORT
======================================================================
Period: 2026-01-15 08:00 to 2026-02-02 10:30
Total queries analyzed: 1523
----------------------------------------------------------------------
QUERY TYPE STATISTICS
----------------------------------------------------------------------
Type Count Avg P25 Min Low%
----------------------------------------------------------------------
general 892 0.712 0.645 0.412 8.2%
mitre_id 456 0.789 0.715 0.521 3.1%
detection 175 0.698 0.621 0.445 12.0%
======================================================================
RECOMMENDATIONS
======================================================================
[1] Lower detection threshold from 0.55 to 0.52
Type: threshold
Confidence: medium
Affected queries: 21
Rationale: 12.0% of 175 queries scored below current threshold.
----------------------------------------------------------------------
APPROVAL REQUIRED
----------------------------------------------------------------------
[1/1] Lower detection threshold from 0.55 to 0.52
Rationale: 12.0% of 175 queries scored below current threshold.
Change: 0.55 -> 0.52
Confidence: medium
Approve? [Y/N/Q]: y
-> APPROVED
1 change(s) applied and saved to tuning_config.json
```
### What Gets Flagged
The attention logger flags these conditions for review:
| Condition | Meaning | Action |
|-----------|---------|--------|
| `zero_results` | No matches found | Content gap - add relevant data |
| `low_score:<score>` | Top score below 0.50 | Poor semantic match |
| `unknown_mitre_ids:<ids>` | MITRE ID not in lookup | Data refresh needed |
| `weak_mitre_match:<score>` | MITRE query scored below 0.60 | Check augmentation |
### Tuning Configuration
Approved changes are saved to `data/tuning_config.json`:
```json
{
"version": "1.0",
"thresholds": {
"general": 0.50,
"mitre_id": 0.55,
"detection": 0.52,
"forensic": 0.55
},
"source_boosts": {
"forensic_clarifications": 1.15
},
"keyword_boost": 1.15,
"last_modified": "2026-02-02T10:45:00",
"last_modified_by": "analyst",
"modification_history": [...]
}
```
The index automatically loads this configuration on startup.
### Safe Bounds
Automated recommendations stay within safe limits:
- Thresholds: 0.40 - 0.70 (never too permissive or restrictive)
- Boosts: 1.0 - 1.30 (maximum 30% boost)
### Audit Trail
The tuning configuration maintains a modification history for accountability:
- Maximum 100 entries retained in `tuning_config.json`
- When limit is exceeded, old entries are logged before removal
- Each entry records: timestamp, parameter, old/new values, approver, reason
For long-term audit retention, configure logging to capture `rag_mcp.tuning_config` INFO messages to a persistent log aggregator.
## Security
### Filesystem Safety
The build system uses sentinel-based deletion guards to prevent accidental data loss:
- Managed directories contain a `.rag_mcp_managed` sentinel file
- **All deletions** go through `safe_rmtree()` with full safety checks
- Sentinel requirement can be relaxed for first-time setup, but all other guards remain
- Forbidden paths (/, /home, /root, etc.) are blocked regardless of sentinel
- Minimum depth checks prevent deleting top-level directories
- Root containment ensures deletions stay within project boundaries
### Network Hardening
All network fetches include multiple security layers:
| Protection | Description |
|------------|-------------|
| **HTTPS-only** | HTTP URLs rejected by default (configurable) |
| **Size limits** | Downloads capped at 60MB to prevent resource exhaustion |
| **Host allowlist** | Only approved hosts (github.com, raw.githubusercontent.com, etc.) |
| **IP literal blocking** | Direct IP addresses blocked to prevent SSRF |
| **Redirect validation** | Redirect targets must be on allowlist |
| **Retry with backoff** | Transient failures (429, 5xx, timeouts) retried with exponential backoff |
### GITHUB_TOKEN Permissions
If you set `GITHUB_TOKEN` for higher API rate limits, use minimal permissions:
```bash
# Create a Fine-grained Personal Access Token with:
# - Public Repositories (read-only) - NO other permissions needed
# - No access to private repositories
# - No write permissions
export GITHUB_TOKEN="github_pat_..."
```
The token is only used for:
- Checking latest commit SHAs (version detection)
- Checking latest release tags
- Cloning public repositories
**Never use tokens with write access or private repo access.**
### Dependency Scanning
Periodically scan dependencies for known vulnerabilities:
```bash
# Install pip-audit
pip install pip-audit
# Scan for vulnerabilities
pip-audit
# Or with requirements file
pip-audit -r requirements.txt
```
Consider adding to CI/CD pipelines for automated scanning.
### Network Exposure Warning
The MCP server uses **stdio transport only** (stdin/stdout). This is secure for local use.
**Do NOT expose this server over a network** without additional hardening:
- No authentication is implemented
- No rate limiting on queries
- No TLS/encryption
If network access is required, place behind an authenticated reverse proxy with rate limiting.
## Requirements
- Python 3.10+
- ~2GB disk space (model + index)
- ~1GB RAM for search operations
## Acknowledgments
Architecture and direction by Steve Anson. Implementation by Claude Code (Anthropic).
## License
MIT
MCP Config
Below is the configuration for this MCP Server. You can copy it directly to Cursor or other MCP clients.
mcp.json
Connection Info
You Might Also Like
No recommendations available