Content
# Tool List
## Overview
HONDANA is a self-hosted RAG (Retrieval-Augmented Generation) system that allows you to accumulate a knowledge base by extracting text from articles, documents, and URLs, and then searching and generating answers using natural language.
## Main Features
- **Multi-format ingestion + extraction fallback** — URLs are fetched and extracted using requests+trafilatura → (JS pages use) Playwright → Jina Reader, in that order. PDFs/Office documents also have text and headings extracted. Reddit posts have text and comments retrieved.
- **3-layer chunking** — L1=article summary / L2=sections (by heading) / L3=paragraphs (with overlap). When searching, hit L3 results are automatically assigned a parent L2 for contextual return.
- **Hybrid search (RRF)** — Vector search and full-text (keyword) search are fused using Reciprocal Rank Fusion. This improves performance on proper nouns and exact matching phrases. Enable with `hybrid: true`.
- **Metadata filtering** — Tags (AND/OR/exclusion), relative dates (within N days), and `source_url` prefix matching for filtering.
- **Recency priority (newer results first)** — Using `recorded_at` for time decay, prioritize newer articles (half-life can be specified). Enabled with `recency_decay_enabled: true`.
- **Results with source** — Each chunk includes `chunk_id` / `section_id` / `section_heading` / `position` / `source_type` / `source_url` / `recorded_at` for accurate citation by agents.
- **MCP Server** — Exposes 10+ tools to agents, including `search_knowledge`. Search filters and saving with duplicate control are also available.
- **Tables (namespaces)** — Collections can be separated by use case (cross-searching is possible).
- **Persistent job queue** — Ingestion is asynchronous, persisted in SQLite (WAL). Jobs are restored after restart, with rate limiting and automatic retries.
- **Duplicate removal, retention, and backup** — Duplicate detection by URL/title/hash, conditional bulk deletion, Qdrant snapshots, and job DB backup scripts.
## Architecture
```
┌──────────── ingest ────────────┐ ┌──────── search ────────┐
URL/file → fetch+extract → chunk(L1/L2/L3) → embed → Qdrant ← vector + FTS ← query
(trafilatura/Playwright/ (headings/paragraphs) (bge-m3 etc.) │ │ ↑ RRF fuse + recency
Jina/PDF/Office/Reddit) │ └→ parent L2 assignment → response generation(LLM)
async job queue (SQLite WAL) ─────────┘
```
- **Vector/metadata storage**: Qdrant (1 table = 1 collection, indexed payload)
- **Job queue**: SQLite (WAL, restarts, retries, rate limiting)
- **Embedding/LLM**: Configurable in `config.yml` with provider/model/URL
- Embedding: Local `sentence-transformers` (default `intfloat/multilingual-e5-small`) or remote OpenAI-compatible endpoints (e.g., vLLM `bge-m3`)
- LLM: Claude/Groq/OpenAI-compatible local (e.g., vLLM, llama.cpp)
## Quick Start
```bash
git clone <repo> hondana && cd hondana
cp .env.example .env # Configure API keys, etc. (if using Claude/Groq)
cp config.example.yml config.yml # Adjust model/URL/search settings
docker compose up -d
# API: http://localhost:8200 (OpenAPI: /docs)
# WebUI: http://localhost:8601
```
## Configuration
### Switching LLM/embedding providers
Change `config.yml` to use different providers/models/URLs. No code changes or rebuilds required.
### Local LLM setup
For local LLM usage, point to your endpoint.
```yaml
# config.yml — Update these 3 lines to your endpoint
llm_answer_provider: openai_compat
llm_answer_model: your-local-model # Model name returned by server (e.g., gemma4-26b / qwen3)
llm_answer_url: http://your-llm-host:8000
```
## Environment Variables (.env)
Secrets and host-specific paths go in `.env` (not version-controlled).
| Variable | Required | Description |
|---|---|---|
| `ANTHROPIC_API_KEY` | For Claude | API key for Claude (response/summary) |
| `GROQ_API_KEY` | For Groq | API key for Groq (summary) |
| `HONDANA_INBOX_DIR` | ✓ | Ingestion source folder (host path) |
| `HONDANA_DONE_DIR` | ✓ | Ingestion destination folder (host path) |
| `HONDANA_API_KEY` | — | WebUI/MCP key. Empty = public |
| `QDRANT_URL` | — | Default `http://qdrant:6333` (compose-internal) |
## Usage
### WebUI
`http://localhost:8601` — Chat search, ingestion, table/article management, settings.
### REST API
Major endpoints:
```
POST /api/ingest/url {url, table, duplicate_action} # Async URL ingestion
POST /api/ingest/file (multipart) # File ingestion
POST /api/ingest/text {title, text, table} # Text ingestion
POST /api/search {query, top_k, hybrid, recency_decay, filters, table}
POST /api/ask {question, ...} # Search + response generation
POST /api/save-answer {question, answer, duplicate_action, agent_id, origin}
GET /api/articles ?table=&limit= # List articles
POST /api/articles/bulk-delete {recorded_days_gte, table} # Conditional bulk deletion
GET /api/jobs/{id} # Job status
```
## Security
**Enable authentication before exposing to the internet.** Default settings are for local/LAN use.
- **Unauthenticated by default**: Empty `api_keys` in `config.yml` allows unrestricted access. Use WebUI settings or set `api_keys` for key-based auth.
## Limitations
- Changing embedding models requires **full re-ingestion** due to compatibility issues.
- Large article counts may slow down listing/sorting due to Qdrant scans.
## Development
```bash
# Install development tools (not a pip package):
pip install ruff black -r api/tests/requirements-test.txt # ruff/black + pytest/pytest-asyncio
pytest # Run tests in api/tests
ruff check api/ && black api/ # Lint/format
```
