Content
# Hyperion Project - Comprehensive Overview
## Executive Summary
**Hyperion** (codebase: `hyper`) is a unified AI-powered code analysis and coordination platform that integrates with Claude Code via the Model Context Protocol (MCP). It provides intelligent code indexing, semantic search, and AI-assisted development workflows through a single Go binary with multiple runtime modes.
### Core Value Proposition
- **Single unified binary** (`hyper`) with three runtime modes
- **AI-powered code understanding** via embeddings and vector search
- **Claude Code integration** through MCP stdio protocol
- **REST API + Web UI** for standalone use
- **Real-time file watching** and automatic code indexing
- **Multi-embedding support** (Ollama, OpenAI, Voyage, TEI)
---
## Project Architecture
### High-Level Overview
```
┌─────────────────────────────────────────────────────────────┐
│ Hyperion (hyper binary) │
├─────────────────────────────────────────────────────────────┤
│ │
│ ┌──────────────┐ ┌──────────────┐ ┌──────────────┐ │
│ │ HTTP Mode │ │ MCP Mode │ │ Both Mode │ │
│ │ (REST + UI) │ │ (stdio) │ │ (default) │ │
│ └──────────────┘ └──────────────┘ └──────────────┘ │
│ │ │ │ │
│ Port 7095 Claude Code Both Active │
│ Web Browser Integration │
│ │
├─────────────────────────────────────────────────────────────┤
│ Core Services │
├─────────────────────────────────────────────────────────────┤
│ │
│ ┌─────────────────────────────────────────────────────┐ │
│ │ Code Indexing & Analysis │ │
│ │ • File watcher (fsnotify) │ │
│ │ • Code parser & tokenizer │ │
│ │ • Semantic indexing │ │
│ └─────────────────────────────────────────────────────┘ │
│ │
│ ┌─────────────────────────────────────────────────────┐ │
│ │ Embedding & Vector Search │ │
│ │ • Multiple embedding providers │ │
│ │ • Qdrant vector database │ │
│ │ • Semantic similarity search │ │
│ └─────────────────────────────────────────────────────┘ │
│ │
│ ┌─────────────────────────────────────────────────────┐ │
│ │ AI Integration │ │
│ │ • LangChain integration │ │
│ │ • Tool definitions (JSON Schema) │ │
│ │ • MCP protocol handlers │ │
│ └─────────────────────────────────────────────────────┘ │
│ │
│ ┌─────────────────────────────────────────────────────┐ │
│ │ Storage Layer │ │
│ │ • MongoDB (metadata, tasks, history) │ │
│ │ • Qdrant (vector embeddings) │ │
│ └─────────────────────────────────────────────────────┘ │
│ │
└─────────────────────────────────────────────────────────────┘
```
### Directory Structure
```
hyper/
├── cmd/
│ └── coordinator/ # Unified binary entry point
│ └── main.go # --mode flag: http|mcp|both
│
├── internal/
│ ├── server/ # HTTP server (Gin framework)
│ │ ├── routes.go # REST API endpoints
│ │ ├── handlers/ # HTTP request handlers
│ │ └── middleware/ # CORS, auth, logging
│ │
│ ├── mcp/ # Model Context Protocol
│ │ ├── handlers/ # MCP tool implementations
│ │ ├── storage/ # MongoDB + Qdrant clients
│ │ ├── embeddings/ # Embedding providers
│ │ ├── indexer/ # Code indexing logic
│ │ ├── watcher/ # File watching
│ │ └── protocol.go # MCP protocol handling
│ │
│ ├── ai-service/
│ │ ├── tools/ # Tool definitions
│ │ └── llm/ # LLM integrations
│ │
│ └── middleware/ # Shared middleware
│
├── embed/ # Embedded UI (auto-generated)
│ └── ui/ # Built React UI
│
├── go.mod # Go dependencies
├── Makefile # Build targets
└── .archived/ # Archived redundant binaries
├── cmd/bridge/
├── cmd/mcp-server/
├── cmd/indexer/
└── cmd/hyper/
coordinator/
└── ui/ # React UI source
├── src/
│ ├── components/ # React components
│ ├── pages/ # Page components
│ ├── services/ # API clients
│ └── App.tsx # Main app
├── dist/ # Built UI (auto-generated)
└── package.json
```
---
## Technology Stack
### Backend (Go)
| Component | Technology | Purpose |
|-----------|-----------|---------|
| **Framework** | Gin Web Framework | HTTP server & routing |
| **Protocol** | MCP Go SDK | Claude Code integration |
| **Database** | MongoDB | Metadata, tasks, history |
| **Vector DB** | Qdrant | Semantic search |
| **File Watching** | fsnotify | Real-time file monitoring |
| **Embeddings** | Multiple providers | Vector generation |
| **Logging** | Uber Zap | Structured logging |
| **LLM Chain** | LangChain Go | AI orchestration |
| **JWT** | golang-jwt | Authentication |
| **WebSocket** | Gorilla WebSocket | Real-time updates |
### Frontend (React)
| Component | Technology | Purpose |
|-----------|-----------|---------|
| **Framework** | React 18+ | UI library |
| **Build Tool** | Vite | Fast bundling |
| **Styling** | TBD | UI styling |
| **API Client** | Fetch/Axios | REST API communication |
| **State** | TBD | State management |
### Embedding Providers
| Provider | Model | Use Case |
|----------|-------|----------|
| **Ollama** (Recommended) | nomic-embed-text | Local, GPU-accelerated, privacy-first (default) |
| **OpenAI** | text-embedding-3-small | Cloud-based, high quality |
| **Voyage AI** | voyage-3 | Specialized embeddings |
| **TEI** | Custom models | Self-hosted embeddings |
> **Recommendation**: We strongly recommend using **Ollama** for embeddings due to:
> - **Privacy**: All code stays on your machine
> - **Cost**: No API fees or rate limits
> - **Performance**: GPU-accelerated local processing
> - **Offline**: Works without internet connection
> - **Quality**: Nomic-embed-text provides excellent code embeddings
### Infrastructure
| Component | Purpose |
|-----------|---------|
| **MongoDB Atlas** | Cloud database |
| **Qdrant Cloud** | Managed vector database |
| **Docker** | Containerization |
| **Docker Compose** | Local development |
---
## Core Features
### 1. **Code Indexing & Analysis**
- **Real-time file watching** using fsnotify
- **Automatic code parsing** and tokenization
- **Semantic indexing** with embeddings
- **Incremental updates** for performance
- **Multi-language support** (Go, Python, JavaScript, etc.)
### 2. **Semantic Search**
- **Vector-based similarity search** via Qdrant
- **Code snippet retrieval** by semantic meaning
- **Context-aware search** using embeddings
- **Filtering and ranking** capabilities
### 3. **Claude Code Integration (MCP)**
- **stdio protocol** for direct Claude integration
- **Tool definitions** in JSON Schema format
- **Real-time code analysis** from Claude
- **Bi-directional communication** with Claude Code
### 4. **REST API + Web UI**
- **RESTful endpoints** for all operations
- **React-based web interface** on port 7095
- **Real-time updates** via WebSocket
- **Authentication** via JWT tokens
- **CORS support** for cross-origin requests
### 5. **File Watching & Auto-Indexing**
- **Recursive directory monitoring**
- **Automatic re-indexing** on file changes
- **Batch processing** for efficiency
- **Configurable watch patterns**
### 6. **AI Service Integration**
- **LangChain integration** for AI workflows
- **Tool calling** for structured AI interactions
- **Prompt templates** for consistent outputs
- **Token counting** and cost estimation
---
## Runtime Modes
### Mode 1: HTTP Mode (`--mode=http`)
```bash
./bin/hyper --mode=http
```
- **REST API** on port 7095
- **Web UI** embedded in binary
- **Standalone operation** without Claude
- **Use case**: Standalone code analysis tool
### Mode 2: MCP Mode (`--mode=mcp`)
```bash
./bin/hyper --mode=mcp
```
- **stdio protocol** for Claude Code
- **No HTTP server** running
- **Direct Claude integration**
- **Use case**: Claude Code plugin
### Mode 3: Both Mode (`--mode=both`) - Default
```bash
./bin/hyper --mode=both
./bin/hyper # Default
```
- **HTTP server** on port 7095
- **MCP stdio** for Claude Code
- **Both interfaces** active simultaneously
- **Use case**: Full-featured development environment
---
## Configuration
### Environment Variables
```bash
# MongoDB
MONGODB_URI="mongodb+srv://user:pass@cluster.mongodb.net"
MONGODB_DATABASE="coordinator_db1"
# Qdrant Vector Database
QDRANT_URL="https://qdrant-instance.com"
QDRANT_KNOWLEDGE_COLLECTION="dev_squad_knowledge"
# Embedding Provider (ollama|openai|voyage|tei)
EMBEDDING="ollama"
# Ollama Configuration
OLLAMA_URL="http://localhost:11434"
OLLAMA_MODEL="nomic-embed-text"
# OpenAI Configuration
OPENAI_API_KEY="sk-..."
OPENAI_MODEL="text-embedding-3-small"
# Voyage AI Configuration
VOYAGE_API_KEY="pa-..."
# Server Configuration
PORT="7095"
LOG_LEVEL="info"
# Code Indexing
CODE_INDEX_AUTO_RECREATE="false"
```
### Configuration File
- **Location**: `.env.hyper` (in executable directory or current directory)
- **Priority**: Custom config path > executable dir > current dir
- **Format**: Standard `.env` format
---
## API Endpoints
### Code Indexing
- `POST /api/index/scan` - Scan directory for code
- `GET /api/index/status` - Get indexing status
- `DELETE /api/index/clear` - Clear all indexed code
### Search
- `POST /api/search/semantic` - Semantic code search
- `GET /api/search/results/:id` - Get search results
### Code Analysis
- `GET /api/code/:fileId` - Get code file
- `POST /api/analyze` - Analyze code snippet
- `GET /api/dependencies/:fileId` - Get file dependencies
### Tasks & History
- `GET /api/tasks` - List tasks
- `POST /api/tasks` - Create task
- `GET /api/history` - Get operation history
### MCP Tools
- `POST /api/mcp/tools` - List available tools
- `POST /api/mcp/execute` - Execute MCP tool
---
## Build & Deployment
### Building
```bash
# Build unified binary with embedded UI
make native
# Development with hot reload
make dev-hot
# Run tests
make test
```
### Output
- **Binary**: `bin/hyper` (~16MB with embedded UI)
- **Platforms**: Linux, macOS, Windows
- **Embedded**: React UI included in binary
### Docker
```bash
# Build Docker image
docker build -t hyperion:latest .
# Run with Docker Compose
docker-compose up
# Run container
docker run -p 7095:7095 \
-e MONGODB_URI="..." \
-e QDRANT_URL="..." \
hyperion:latest
```
---
## Use Cases
### 1. **AI-Assisted Code Review**
- Analyze code changes with AI
- Get semantic understanding of code
- Identify patterns and issues
### 2. **Claude Code Integration**
- Use as Claude Code plugin
- Real-time code analysis in Claude
- Semantic search from Claude
### 3. **Code Search & Navigation**
- Find similar code patterns
- Discover related files
- Navigate large codebases
### 4. **Documentation Generation**
- Auto-generate docs from code
- Create API documentation
- Generate architecture diagrams
### 5. **Code Quality Analysis**
- Detect code smells
- Identify refactoring opportunities
- Enforce coding standards
### 6. **Knowledge Management**
- Index project knowledge
- Store architectural decisions
- Maintain code documentation
---
## Development Workflow
### Setup
```bash
# Install dependencies
make install
# Install Air for hot reload
make install-air
# Configure environment
cp .env.example .env.hyper
```
### Development
```bash
# Start with hot reload (Go + UI)
make dev-hot
# Or just Go hot reload
make dev
# Run tests
make test
# Build for distribution
make native
```
### Testing
```bash
# Run all tests
make test
# Run specific test
go test ./internal/mcp/handlers -v
# Test with coverage
go test -cover ./...
```
---
## Key Components Deep Dive
### Code Indexer
- **Location**: `internal/mcp/indexer/`
- **Purpose**: Parse and index code files
- **Features**:
- Language detection
- Token extraction
- Function/class identification
- Dependency analysis
### Embedding Service
- **Location**: `internal/mcp/embeddings/`
- **Purpose**: Generate vector embeddings
- **Providers**:
- Ollama (local, GPU)
- OpenAI (cloud)
- Voyage AI (specialized)
- TEI (self-hosted)
### Storage Layer
- **Location**: `internal/mcp/storage/`
- **Components**:
- MongoDB client (metadata)
- Qdrant client (vectors)
- Collection management
- Query builders
### MCP Handlers
- **Location**: `internal/mcp/handlers/`
- **Purpose**: Implement MCP tools
- **Tools**:
- Code analysis
- Search
- Indexing
- File operations
### HTTP Server
- **Location**: `internal/server/`
- **Framework**: Gin Web Framework
- **Features**:
- RESTful routing
- Middleware (CORS, auth)
- Error handling
- Request validation
---
## Performance Characteristics
### Indexing
- **Speed**: ~1000 files/second (depends on file size)
- **Memory**: ~100MB for 10K files
- **Storage**: ~1MB per 1000 files (metadata)
### Search
- **Latency**: <100ms for semantic search
- **Throughput**: 100+ queries/second
- **Accuracy**: High (vector-based similarity)
### API
- **Response Time**: <50ms for most endpoints
- **Throughput**: 1000+ requests/second
- **Concurrency**: Fully concurrent
---
## Security Considerations
### Authentication
- **JWT tokens** for API access
- **Token expiration** and refresh
- **Role-based access control** (RBAC)
### Data Protection
- **Encryption in transit** (HTTPS)
- **Encryption at rest** (MongoDB)
- **API key management** for external services
### Code Privacy
- **Local indexing** option (Ollama)
- **No code sent to external services** (unless configured)
- **Configurable data retention**
---
## Troubleshooting
### Vector Dimension Mismatch
**Problem**: Switching embedding models causes dimension mismatch
**Solution**:
```bash
# Auto-recreate collection
export CODE_INDEX_AUTO_RECREATE=true
./bin/hyper --mode=http
# Or manually confirm when prompted
```
### MongoDB Connection Issues
**Problem**: Cannot connect to MongoDB
**Solution**:
```bash
# Verify connection string
echo $MONGODB_URI
# Test connection
mongosh "$MONGODB_URI"
```
### Qdrant Connection Issues
**Problem**: Cannot connect to Qdrant
**Solution**:
```bash
# Check Qdrant health
curl https://your-qdrant-url/health
# Verify URL in config
echo $QDRANT_URL
```
### Embedding Service Issues
**Problem**: Embedding generation fails
**Solution**:
```bash
# For Ollama (Recommended): ensure service is running
brew services start ollama # macOS
# or
systemctl start ollama # Linux with systemd
# Pull the model if not already available
ollama pull nomic-embed-text
# Test embedding generation
curl http://localhost:11434/api/embeddings -d '{
"model": "nomic-embed-text",
"prompt": "test"
}'
# For OpenAI: verify API key
echo $OPENAI_API_KEY
# For Voyage AI: verify API key
echo $VOYAGE_API_KEY
```
**See the [Using Ollama for Embeddings](#using-ollama-for-embeddings-recommended) section for detailed setup and troubleshooting.**
---
## Project Status
### ✅ Completed
- [x] Unified binary architecture
- [x] HTTP + MCP modes
- [x] Code indexing
- [x] Vector search
- [x] REST API
- [x] Web UI
- [x] MongoDB integration
- [x] Qdrant integration
- [x] Multiple embedding providers
- [x] File watching
- [x] MCP protocol support
### 🚀 In Development
- [ ] Advanced code analysis
- [ ] Refactoring suggestions
- [ ] Architecture visualization
- [ ] Performance optimization
### 📋 Planned
- [ ] Desktop application
- [ ] IDE plugins (VS Code, JetBrains)
- [ ] Git integration
- [ ] CI/CD integration
- [ ] Team collaboration features
---
## Contributing
### Code Style
- Follow Go conventions
- Use `gofmt` for formatting
- Add tests for new features
- Document public APIs
### Testing
```bash
# Run all tests
make test
# Run specific package
go test ./internal/mcp/handlers -v
# With coverage
go test -cover ./...
```
### Building
```bash
# Clean build
make clean && make native
# Verify binary
./bin/hyper --version
```
---
## License
[Add your license information here]
---
## Support & Resources
### Documentation
- **README**: This file (comprehensive overview)
- **OLLAMA_SETUP_GUIDE.md**: Ollama installation and embedding model selection
- **CLEAN_INSTALL_GUIDE.md**: Clean installation guide
- **CLEAN_INSTALL_COMPLETE.md**: Clean install implementation details
- **CLEANUP_COMPLETE.md**: Build system details
- **MAKEFILE_CLEANUP_SUMMARY.md**: Makefile reference
### Getting Help
- Check troubleshooting section above
- Review environment variables
- Check logs for errors
### Community
- [GitHub Issues](https://github.com/your-repo/issues)
- [Discussions](https://github.com/your-repo/discussions)
---
## Quick Start
### 1. Install Dependencies
```bash
make install
```
### 2. Configure Environment
```bash
cp .env.example .env.hyper
# Edit .env.hyper with your settings
```
### 3. Build
```bash
make native
```
### 4. Run
```bash
# HTTP mode with UI
./bin/hyper --mode=http
# MCP mode for Claude
./bin/hyper --mode=mcp
# Both modes (default)
./bin/hyper
```
### 5. Access
- **Web UI**: http://localhost:7095
- **API**: http://localhost:7095/api
- **Claude Code**: Configure with `make configure-native`
---
## Using Ollama for Embeddings (Recommended)
### Why Ollama?
Ollama is our **recommended embedding provider** for Hyperion because it offers:
- **🔒 Privacy-First**: Your code never leaves your machine
- **💰 Zero Cost**: No API fees or rate limits
- **⚡ Fast**: GPU-accelerated local processing
- **📴 Offline**: Works without internet connection
- **🎯 High Quality**: Nomic-embed-text model is optimized for code embeddings
- **🚀 Easy Setup**: Simple installation and configuration
### Installation
#### macOS
```bash
# Install via Homebrew
brew install ollama
# Start Ollama service
brew services start ollama
# Verify installation
ollama --version
```
#### Linux
```bash
# Install via curl
curl -fsSL https://ollama.com/install.sh | sh
# Start Ollama service
ollama serve &
# Verify installation
ollama --version
```
#### Windows
```powershell
# Download installer from https://ollama.com/download
# Run the installer and follow instructions
# Ollama will start automatically
```
### Setup for Hyperion
#### 1. Pull the Embedding Model
```bash
# Pull the recommended nomic-embed-text model
ollama pull nomic-embed-text
# Verify model is available
ollama list
```
Expected output:
```
NAME ID SIZE MODIFIED
nomic-embed-text:latest a80c4f17acd5 274MB 2 minutes ago
```
#### 2. Configure Hyperion
Edit your `.env.hyper` file:
```bash
# Embedding Provider Configuration
EMBEDDING=ollama
# Ollama Configuration
OLLAMA_URL=http://localhost:11434
OLLAMA_MODEL=nomic-embed-text
# Embedding Dimensions (must match model)
EMBEDDING_DIMENSION=768
```
#### 3. Verify Ollama is Running
```bash
# Test Ollama API
curl http://localhost:11434/api/tags
# Test embedding generation
curl http://localhost:11434/api/embeddings -d '{
"model": "nomic-embed-text",
"prompt": "test code snippet"
}'
```
#### 4. Start Hyperion
```bash
# Build and run
make native
./bin/hyper --mode=http
# Or with hot reload
make dev-hot
```
### Usage Examples
#### Example 1: Index Your Codebase
```bash
# Start Hyperion with Ollama
./bin/hyper --mode=http
# Open browser to http://localhost:7095
# Navigate to Code Search
# Click "Add Folder" and select your project directory
# Ollama will generate embeddings locally
```
#### Example 2: Semantic Search
```bash
# Search for authentication code
curl -X POST http://localhost:7095/api/search/semantic \
-H "Content-Type: application/json" \
-d '{
"query": "user authentication with JWT tokens",
"limit": 10
}'
```
#### Example 3: Programmatic Usage
```go
package main
import (
"github.com/ollama/ollama/api"
)
func main() {
client, _ := api.ClientFromEnvironment()
// Generate embeddings for code
req := &api.EmbeddingRequest{
Model: "nomic-embed-text",
Prompt: "function calculateTotal(items) { return items.reduce((a,b) => a+b, 0); }",
}
resp, _ := client.Embeddings(context.Background(), req)
// resp.Embedding contains 768-dimensional vector
}
```
### Advanced Ollama Configuration
#### Using Different Models
```bash
# Try other embedding models
ollama pull mxbai-embed-large # 335M params, 1024 dimensions
ollama pull all-minilm # 22M params, 384 dimensions
# Update .env.hyper
OLLAMA_MODEL=mxbai-embed-large
EMBEDDING_DIMENSION=1024
```
#### GPU Acceleration
Ollama automatically uses GPU if available:
```bash
# Check GPU usage
nvidia-smi # NVIDIA GPUs
# or
metal-smi # Apple Silicon
```
#### Performance Tuning
```bash
# Adjust Ollama settings for performance
# In ~/.ollama/config.json
{
"num_gpu": 1, # Number of GPUs to use
"num_thread": 8, # CPU threads
"num_parallel": 4 # Parallel requests
}
```
### Troubleshooting Ollama
#### Ollama Service Not Running
```bash
# macOS
brew services restart ollama
# Linux
killall ollama && ollama serve &
# Check status
curl http://localhost:11434/api/tags
```
#### Model Not Found
```bash
# Re-pull the model
ollama pull nomic-embed-text
# Verify it's available
ollama list
```
#### Dimension Mismatch Error
```bash
# Hyperion will detect dimension mismatch and offer to recreate collection
# Or manually set auto-recreate:
export CODE_INDEX_AUTO_RECREATE=true
./bin/hyper --mode=http
```
#### Slow Embedding Generation
```bash
# Ensure GPU is being used
ollama ps # Should show GPU memory usage
# If CPU-only, check GPU drivers
nvidia-smi # NVIDIA
system_profiler SPDisplaysDataType # macOS
```
### Migrating from Other Providers
#### From OpenAI to Ollama
```bash
# 1. Install and configure Ollama (see above)
# 2. Update .env.hyper
EMBEDDING=ollama
OLLAMA_URL=http://localhost:11434
OLLAMA_MODEL=nomic-embed-text
EMBEDDING_DIMENSION=768
# 3. Recreate code index (dimensions changed from 1536 to 768)
export CODE_INDEX_AUTO_RECREATE=true
./bin/hyper --mode=http
# 4. Re-index your code
# The system will automatically use Ollama for new embeddings
```
#### From Voyage to Ollama
```bash
# Similar process, just update EMBEDDING variable
EMBEDDING=ollama
# Rest of the steps are the same
```
### Cost Comparison
| Provider | Cost per 1M tokens | Notes |
|----------|-------------------|-------|
| **Ollama** | **FREE** | Unlimited local usage |
| OpenAI | ~$0.13 | text-embedding-3-small |
| Voyage AI | ~$0.12 | voyage-3 |
| TEI | Infrastructure costs | Self-hosted |
For a typical medium-sized codebase (10K files), you might generate 50M tokens of embeddings, which would cost ~$6.50 with cloud providers but is **completely free** with Ollama.
---
## Architecture Highlights
### Single Binary Approach
- **One executable** with all features
- **No separate services** needed
- **Easy deployment** and distribution
- **Reduced complexity** and maintenance
### Modular Design
- **Clear separation of concerns**
- **Pluggable components** (embeddings, storage)
- **Easy to extend** and customize
- **Testable architecture**
### Cloud-Ready
- **MongoDB Atlas** for scalability
- **Qdrant Cloud** for vector search
- **Docker support** for containerization
- **Environment-based configuration**
---
*Last Updated: 2024*
*Project: Hyperion (hyper)*
