opencrawling

# OpenCrawling [](LICENSE) [](https://jdk.java.net/25/) [](https://spring.io/projects/spring-boot) [](https://www.docker.com/) [](https://kafka.apache.org/) [](https://www.postgresql.org/) [](https://redis.io/) [](https://github.com/OpenPj/spring-manifold-next-gen/stargazers) [](https://github.com/OpenPj/spring-manifold-next-gen/issues) [](https://github.com/OpenPj/spring-manifold-next-gen/pulls) **OpenCrawling** is an enterprise data integration and ingestion platform modeled after Apache ManifoldCF. It leverages modern Java 25 features (such as Structured Concurrency and Virtual Threads), Spring Boot, and vector search infrastructure to orchestrate data flows from various repository connectors to vector search outputs. <p align="center"> <img src="https://github.com/opencrawling/.github/raw/main/profile/images/logo.png" alt="OpenCrawling Logo" width="200" /> </p> --- ## Architecture Diagram The diagram below shows the high-level architecture of OpenCrawling: ```mermaid graph TD subgraph UI UI_App[Admin React UI - oc-admin-ui] end subgraph Platform Runtime [OpenCrawling JVM Runtime] Core[Core Ingestion Engine - oc-core] Runtime[Bootstrap - oc-runtime] FS_Conn[Filesystem Repository - oc-filesystem-repository-connector] Ing_Cons[Ingestion Consumer - IngestionConsumer] Tika[Apache Tika Text Extractor] Chunker[Token Chunker] Embed_Cons[Embedding Consumer - EmbeddingConsumer] Writer_Cons[Vector Store Writer - VectorStoreWriterConsumer] Precompute_Model[PrecomputedEmbeddingModel] Vec_Conn[Vector Store Output - oc-vector-output-connector] McpServer[Secure MCP Server - McpVectorServer] Runtime --> Core Core --> FS_Conn Core -->|Publish IngestionMessage| Ingest_Topic[(Kafka Topic: opencrawling-ingestion)] Ingest_Topic -->|Consume IngestionMessage| Ing_Cons Ing_Cons -->|Extract Text| Tika Tika --> Chunker Chunker -->|Publish Chunks| Chunk_Topic[(Kafka Topic: opencrawling-chunks)] Chunk_Topic -->|Consume ChunkMessage| Embed_Cons Embed_Cons -->|Publish Embedded| Embed_Topic[(Kafka Topic: opencrawling-embedded)] Embed_Topic -->|Consume EmbeddedMessage| Writer_Cons Writer_Cons --> Precompute_Model Precompute_Model --> Vec_Conn McpServer -->|Queries - Enforces ACLs| Vec_Conn end subgraph Infrastructure [Docker Containers] PG[(PostgreSQL + pgvector)] Redis[(Redis Cache & Session)] Ollama[Ollama AI Embeddings] Kafka_Broker[Apache Kafka Broker] end subgraph External [AI Clients] LLM[AI Client / LLM Agent] end UI_App -->|REST API| Runtime Vec_Conn -->|Vectors| PG Runtime -->|Job Cache| Redis Embed_Cons -->|Generates Embeddings| Ollama Ingest_Topic --> Kafka_Broker Chunk_Topic --> Kafka_Broker Embed_Topic --> Kafka_Broker LLM -->|Model Context Protocol| McpServer ``` --- ## Administration Dashboard (oc-admin-ui) The `oc-admin-ui` provides a modern web-based administration console to monitor and configure your ingestion jobs. ### UI Screenshots #### 📊 Telemetry Dashboard  *Real-time graphs monitoring job success rates, Kafka queue load, active crawling threads, and index ingestion speed.* #### 📋 Job Pipeline Scheduler  *Schedule, monitor, start, and pause ingestion crawl tasks. Review document indexing status reports.* #### 📁 Connector Configurations  *Manage endpoints and credentials for repositories (SharePoint, S3, Filesystem) and vector search destinations.* #### ⚙️ Ingestion & Embedding Mappings  *Configure target models (e.g., Ollama, OpenAI) and tune text chunk sizes/overlap boundaries dynamically.* #### 🪵 Real-Time Ingestion Logs  *Inspect live Java logging streams and Kafka consumer offsets to troubleshoot connector execution.* --- ## Core Technologies - **Java 25 Preview Features**: Structured Concurrency, Virtual Threads, and Pattern Matching. - **Spring Boot & Spring AI**: High-performance backend orchestrating ingestion jobs. - **Apache Kafka**: Decoupled, event-driven document processing using the **Claim Check Pattern**. - **pgvector**: High-dimensional vector similarity search in PostgreSQL. - **Redis Stack**: Lightweight caching and session management. - **Ollama**: Local AI embedding generation via open-source LLM models. - **Vite + React + TailwindCSS**: Modern frontend administration dashboard. --- ## Getting Started ### Prerequisites Ensure you have the following installed on your machine: - **JDK 25** (Ensure `JAVA_HOME` points to your JDK 25 directory) - **Maven 3.9+** - **Docker & Docker Compose** - **Node.js 18+ & npm** (for the UI) --- ### Step-by-Step Setup #### 1. Start Infrastructure (Docker) Spin up the database, cache, message broker, and AI engine. Run from the project root: ```bash docker compose up -d ``` **Services started:** * **PostgreSQL (Port 5432)**: For job metadata, schema migrations, and pgvector storage. * **Redis (Port 6379 / Insight Port 8001)**: For caching and session management. * **Ollama (Port 11434)**: For local embeddings. * **Apache Kafka (Port 9092)**: KRaft-mode broker for decoupled, event-driven document processing (internal container communication on port `9094`). #### 2. Pull the Embedding Models (Ollama) OpenCrawling supports configuring different embedding models on a per-job basis and automatically routes them to corresponding PgVector tables. To use the available options, make sure to pull the models you plan to utilize: * **mxbai-embed-large (1024-dim, default)**: ```bash docker exec -it ollama ollama pull mxbai-embed-large ``` * **nomic-embed-text (768-dim)**: ```bash docker exec -it ollama ollama pull nomic-embed-text ``` * **all-minilm (384-dim)**: ```bash docker exec -it ollama ollama pull all-minilm ``` *(Ollama will download the requested models in the background. Once pulled, OpenCrawling will automatically route them to `vector_store_1024`, `vector_store_768`, or `vector_store_384` respectively).* --- ### Option A: Run OpenCrawling in Docker Containers (Recommended) To build and run the OpenCrawling backend runtime and administration UI as containerized services, run: 1. **Build the images**: ```bash docker compose -f docker-compose-apps.yml build ``` 2. **Start the applications**: ```bash docker compose -f docker-compose-apps.yml up -d ``` * **Backend Service**: Access the backend runtime and integrated static resources at [http://localhost:8080](http://localhost:8080). * **Frontend Service**: Access the standalone React Administration Console at [http://localhost:3000](http://localhost:3000). --- ### Option A.2: Decoupled Multi-Service Deployment (Decoupled Microservices) To run each microservice component (Repository Crawler, Ingestion Consumer, Embedding Consumer, Vector Store Writer, Secure MCP Server, and Admin UI) as a completely separate containerized process communicating over Kafka: 1. **Build the decoupled service images**: ```bash docker compose -f docker-compose-decoupled.yml build ``` 2. **Start the complete decoupled pipeline**: ```bash docker compose -f docker-compose-decoupled.yml up -d ``` This spins up the database/event-stream dependencies alongside five decoupled OpenCrawling service containers. You can view logs, scale individual workers (e.g. `docker compose -f docker-compose-decoupled.yml scale oc-embedding-consumer=3`), and monitor the decoupled pipeline. * **React Admin UI Console**: Access the administration dashboard at [http://localhost:3000](http://localhost:3000). * **Secure MCP Server**: Connect your AI Client / IDE directly to [http://localhost:8080](http://localhost:8080) over SSE. #### Running the Decoupled Integration Test We provide a fully automated end-to-end integration test script that builds, boots, tests, and cleanses the entire decoupled environment: ```bash ./scripts/test-docker-decoupled.sh ``` This script: 1. Builds all decoupled microservices from source. 2. Boots up the Kafka broker, PostgreSQL + pgvector, Redis, Ollama, and consumer workers. 3. Automatically generates a sample document in the crawler mount, triggers a scan, and waits for consumer ingestion. 4. Queries pgvector directly to verify that the generated embeddings are correctly stored. 5. Verifies the Secure MCP Server SSE endpoint, and tears down the environment upon success. --- ### Option B: Run OpenCrawling Locally (Development Mode) If you wish to run the JVM runtime and React frontend directly on your host machine for development: #### 1. Build the Project (Maven) Compile all modules using Java 25. Since we utilize advanced features, preview features must be enabled: ```bash mvn clean install ``` #### 2. Run the Runtime Bootstrap Start the Spring Boot runtime application: ```bash mvn spring-boot:run -pl oc-runtime -Dspring-boot.run.profiles=dev ``` ##### Running a Sample Ingestion Job on Startup (Optional) By default, the automatic startup crawl is disabled to prevent unnecessary scans. To trigger a demo crawl job on startup, pass the configuration properties: ```bash mvn spring-boot:run -pl oc-runtime -Dspring-boot.run.profiles=dev \ -Dspring-boot.run.arguments="--spring.opencrawling.crawl-on-startup=true --spring.opencrawling.scan-path=/your/local/directory/to/scan" ``` #### 3. Run the Admin UI To launch the administration dashboard: ```bash cd oc-admin-ui npm install npm run dev ``` Open [http://localhost:5173](http://localhost:5173) in your browser. --- ## Scaling Out & Performance OpenCrawling is designed for high-throughput, horizontal scalability. Since the ingestion pipeline is decoupled using **Apache Kafka** and the **Claim Check Pattern**, you can scale components independently. ### 1. Scaling the Ingestion / Processing (Output Connector) Vector indexing and embedding generation is typically the primary performance bottleneck because of deep learning model inference (Ollama) and database indexing (pgvector). * **Kafka Consumer Group Partitioning**: The three main topics (`opencrawling-ingestion`, `opencrawling-chunks`, and `opencrawling-embedded`) are consumed by `IngestionConsumer`, `EmbeddingConsumer`, and `VectorStoreWriterConsumer` respectively within the `oc-runtime` service. By configuring these topics with multiple partitions, Kafka distributes load dynamically among active consumer nodes. * **Horizontal Scaling of Runtime Instances**: You can run multiple instances of the `oc-runtime` application sharing the same `spring.application.name` and consumer group (`opencrawling-vector-group`). Kafka automatically distributes partitions and load-balances the messages. * **Ollama Load Balancing**: Scale out embedding generation by pointing `spring.ai.ollama.base-url` to a load balancer (e.g., NGINX, HAProxy) backed by a cluster of Ollama instances running on GPU-enabled nodes. ### 2. Scaling the Repository Connectors (Ingestion Source) The scanning/crawling phase can be distributed by splitting large target sources: * **Partitioned Scans**: Run separate bootstrap crawl jobs targeting different sub-directories or repository prefixes. * **Distributed File Shares / Shared Storage**: In a multi-node setup, ensure the `IngestionConsumer` instances have access to the same shared filesystem (e.g., NFS, S3/MinIO bucket, SMB) as the repository crawlers, so the Claim Check reference (path/URI) can be successfully resolved by the consumer node. ### 3. Claim Check Pattern To ensure the messaging system remains fast and responsive: 1. The **Repository Connector** crawls data, but instead of publishing the entire document content (which could be megabytes of binary data) to Kafka, it saves/references the file on a shared storage medium. 2. It publishes a lightweight `IngestionMessage` (Claim Check record) to the Kafka topic containing the metadata (URI, file path, version). 3. The **Consumer Workers** process the ingestion: * **`IngestionConsumer`** pulls the reference, reads the file directly from storage, extracts text with **Apache Tika**, splits it into semantic chunks, and publishes them to the chunks topic. * **`EmbeddingConsumer`** pulls the chunks, requests embedding vectors from **Ollama**, and publishes the embedded chunks to the embedded topic. * **`VectorStoreWriterConsumer`** consumes embedded chunks and uses a `PrecomputedEmbeddingModel` to save them directly to pgvector. --- ## Verification & Monitoring - **Database**: Access PostgreSQL at `localhost:5432` (User: `opencrawling`, DB: `opencrawling`). - **Redis Dashboard**: Open [http://localhost:8001](http://localhost:8001) in your browser to view the Redis Stack Insight dashboard. - **Logs**: Monitor console output for the Virtual Thread Executor and Structured Concurrency task logs. --- ## Troubleshooting - **Java Version Check**: Run `java -version` to confirm you are using Java 25. - **Preview Features**: If your IDE fails to compile structured concurrency code, verify that the `--enable-preview` JVM argument is configured for compiler and runtime settings. (It is already pre-configured in `pom.xml`). --- ## Trademark OpenCrawling&reg; is a registered trademark of the OpenCrawling Organization. For guidelines on using the name and logo, please refer to the [TRADEMARK.md](TRADEMARK.md) file.